Logistic Regression from Scratch

Logistic Regression from Scratch

Let’s us try to implement logistic regression from scratch in python.

Recommended to be read after the Neural Networks release.

To test and run the code used check out: https://www.kaggle.com/code/tanavbajaj/logistic-regression-math-behind-without-sklearn

Importing necessary libraries

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib as plt

The dataset we will be using is pima-indians-diabetes-database.

Whose objective is to diagnostically predict whether or not a patient has diabetes.

y = data.Outcome.values
x_data = data.drop([“Outcome”],axis=1)

After this the data is divided into y which is the desired classification value column and x_data which refers to the various feature of the dataset

Normalization of the data

The value of column DiabetesPedigreeFunction varies from 0.08 to 2.48 while Insulin varies from 0 to 848. We normalize data to give equal weightage to both the columns

x = (x_data – np.min(x_data)) / (np.max(x_data) – np.min(x_data)).values

Upon normalization the data is converted to the range of 0 – 1.

After this data is split into training and testing datasets

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(x,y,test_size = 0.25, random_state = 42)

x_train = x_train.T
x_test = x_test.T
y_train = y_train.T
y_test = y_test.T

We do this using the built-in method inside the sklearn library.

Defining Necessary Functions

Initialize the weights and biases

The weights and biases are called the parameter of the model. Each feature in the training dataset is given a certain weight. Let’s start by assigning them all some random value i.e. 0.01.

def initialize_weights_and_bias(dimension):

w = np.full((dimension,1),0.01)
b = 0.0
return w,b

Sigmoid function

Logistic regression ,uses regression to predict the label by making a linear decision boundary. It would make no sense to get the value greater than 1 or less than 0. Here the sigmoid function comes into play. This function ranges between 0 to 1 and is stated as

def sigmoid(z):

y_head = 1 / (1+np.exp(-z))

return y_head

To get the value of the parameter z used in the sigmoid function we use the formula  where x is the features array, w weights and b is the bias. Now to make our model learn we need to punish it for the losses and penalize it for wrong predictions this is done by the loss function which is stated as

And cost function is effectively the sum of loss across all the cases.

This process of penalizing the model and moving forward with the predictions is known as forward propagation.

If we recall we allotted random weights to the parameter, they now will be updated based upon our loss function and cost function. To minimize loss and cost function we use gradient descent which is

Where w denotes the weights,  denotes the stepsize or the factor by which to change the gradient to find the local minima which are multiplied by the derivative of the loss function to sum it all up the algorithm works as follow:-

We assume a random datapoint in our graph and calculate its slope, then we find the direction in which the loss function decreases and update the weights using the gradient descent formula this process is known as back propagation, we then select point by taking a stepsize of  and repeat this entire process once again

sometimes is also referred to as learning rate.

Finally we write the code for forward and backward propagation combined as backward propagation also uses the same z which is found in forward propagation

def forward_backward_propagation(w,b,x_train,y_head):

z = np.dot(w.T,x_train) + b
y_head = sigmoid(z)
loss = -y_train*np.log(y_head) – (1-y_train)*np.log(1-y_head)
cost = (np.sum(loss)) / x_train.shape[1]

#backward propogation
derivative_weight = (np.dot(x_train,((y_head-y_train).T)))/x_train.shape[1]
derivative_bias = np.sum(y_head-y_train)/x_train.shape[1]
gradients = {“derivative_weight”: derivative_weight,”derivative_bias”: derivative_bias}
return cost,gradients

After we have calculated our parameters we need to update the randomly assigned weights to do this we use another update function

def update(w, b, x_train, y_train, learning_rate,number_of_iterarion):
cost_list = []
cost_list2 = []
index = []
# updating(learning) parameters is number_of_iterarion times
for i in range(number_of_iterarion):
# make forward and backward propagation and find cost and gradients
cost,gradients = forward_backward_propagation(w,b,x_train,y_train)
# lets update
w = w – learning_rate * gradients[“derivative_weight”]
b = b – learning_rate * gradients[“derivative_bias”]
if i % 10 == 0:
print (“Cost after iteration %i: %f” %(i, cost))
parameters = {“weight”: w,”bias”: b}return parameters, gradients, cost_list

After this function is called our model has successfully calculated the values of weights and biases using the method of forward propagation and backward propagation. This process is also used in neural network i.e. they repeatedly update weights using forward and backward propagation.

Now onto the most awaited part of generating the predictions we do that by defining a predict function as follow

def predict(w,b,x_test):
# x_test is a input for forward propagation
z = sigmoid(np.dot(w.T,x_test)+b)
Y_prediction = np.zeros((1,x_test.shape[1]))
# if z is bigger than 0.5, our prediction is one means has diabete (y_head=1),
# if z is smaller than 0.5, our prediction is zero means does not have diabete (y_head=0),
for i in range(z.shape[1]):
if z[0,i]<= 0.5:
Y_prediction[0,i] = 0
Y_prediction[0,i] = 1return Y_prediction

Combining all the functions

def logistic_regression(x_train, y_train, x_test, y_test, learning_rate ,  num_iterations):
# initialize
dimension =  x_train.shape[0]
w,b = initialize_weights_and_bias(dimension)parameters, gradients, cost_list = update(w, b, x_train, y_train, learning_rate,num_iterations)

y_prediction_test = predict(parameters[“weight”],parameters[“bias”],x_test)

# Print train/test Errors
print(“test accuracy: {} %”.format(100 – np.mean(np.abs(y_prediction_test – y_test)) * 100))


from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
print(“Test Accuracy {}”.format(lr.score(x_test.T,y_test.T)))

That was all about Logistic Regression without inbuilt python libraries (sklearn).

Leave a Reply

Your email address will not be published. Required fields are marked *