Linear Regression with Gradient Descent and Python
Gradient Descent is the key optimization method used in machine learning. Understanding how gradient descent works without using API helps to gain a deep understanding of machine learning. This article will demonstrates how you can solve linear regression problem using gradient descent method.
Our test data (x,y) is shown below. It is a simple linear function 2*x+3 with some random noise
import numpy as np import matplotlib.pyplot as plt x = np.linspace(0,10,20) y = 2*x+3+2*np.random.rand(len(x)) plt.scatter(x,y) plt.show()
We use a model given by yhat = w*x +b. The loss function is a Mean Square Error (MSE) given by the mean sum of (yhat-y)**2. We compute the gradients of the loss function for w and b, and then update the w and b for each iteration. We output the final w, b, as well as the loss in each iteration.
def gradientDescent(x, y, theta, learn_rate, N, n_iter): loss_i = np.zeros(n_iter) for i in range(n_iter): w = theta b = theta yhat = w*x+b loss = np.sum((yhat-y)** 2)/(2*N) loss_i[i] = loss print("i:%d, loss: %f" % (i, loss)) gradient_w = np.dot(x,(yhat-y))/N gradient_b = np.sum((yhat-y))/N w = w - learn_rate*gradient_w b = b - learn_rate*gradient_b theta = [w,b] return theta,loss_i
We set the hyperparametrs and run the gradient descent to determine the best w and b
n_iter = 100 learn_rate = 0.05 theta = np.zeros(2) N = len(x) theta,loss = gradientDescent(x, y, theta, learn_rate, N, n_iter) print(theta)
After the iteration, we plot of the best fit line overlay to the raw data as shown below
import matplotlib.pyplot as plt plt.plot(x,y,'or') plt.plot(x,theta*x+theta,'b') plt.show()
We also plot the loss as a function of iteration. The loss reduces with time indicating the model is learning to fit to the data points
import matplotlib.pyplot as plt epoch = range(len(loss)) plt.plot(epoch,loss,'o') plt.xlabel('iteration') plt.ylabel('loss') plt.show()