## Linear Regression with Gradient Descent and Python

Gradient Descent is the key optimization method used in machine learning. Understanding how gradient descent works without using API helps to gain a deep understanding of machine learning. This article will demonstrates how you can solve linear regression problem using gradient descent method.

Our test data (x,y) is shown below. It is a simple linear function 2*x+3 with some random noise

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0,10,20)
y = 2*x+3+2*np.random.rand(len(x))

plt.scatter(x,y)
plt.show()

We use a model given by yhat = w*x +b. The loss function is a Mean Square Error (MSE) given by the mean sum of (yhat-y)**2. We compute the gradients of the loss function for w and b, and then update the w and b for each iteration. We output the final w, b, as well as the loss in each iteration.

def gradientDescent(x, y, theta, learn_rate, N, n_iter):
loss_i = np.zeros(n_iter)
for i in range(n_iter):
w = theta[0]
b = theta[1]
yhat = w*x+b
loss = np.sum((yhat-y)** 2)/(2*N)
loss_i[i] = loss
print("i:%d, loss: %f" % (i, loss))

w = w - learn_rate*gradient_w
b = b - learn_rate*gradient_b
theta = [w,b]
return theta,loss_i


We set the hyperparametrs and run the gradient descent to determine the best w and b

n_iter = 100
learn_rate = 0.05
theta = np.zeros(2)
N = len(x)

theta,loss = gradientDescent(x, y, theta, learn_rate, N, n_iter)
print(theta)

After the iteration, we plot of the best fit line overlay to the raw data as shown below

import matplotlib.pyplot as plt

plt.plot(x,y,'or')
plt.plot(x,theta[0]*x+theta[1],'b')
plt.show()

We also plot the loss as a function of iteration. The loss reduces with time indicating the model is learning to fit to the data points

import matplotlib.pyplot as plt

epoch = range(len(loss))
plt.plot(epoch,loss,'o')
plt.xlabel('iteration')
plt.ylabel('loss')
plt.show()

May 3, 2021