Let’s us try to implement logistic regression from scratch in python.
Recommended to be read after the Neural Networks release.
To test and run the code used check out: https://www.kaggle.com/code/tanavbajaj/logistic-regression-math-behind-without-sklearn
Importing necessary libraries
| import numpy as np # linear algebra import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv) import matplotlib as plt |
The dataset we will be using is pima-indians-diabetes-database.
Whose objective is to diagnostically predict whether or not a patient has diabetes.
| y = data.Outcome.values x_data = data.drop([“Outcome”],axis=1) |
After this the data is divided into y which is the desired classification value column and x_data which refers to the various feature of the dataset
Normalization of the data
The value of column DiabetesPedigreeFunction varies from 0.08 to 2.48 while Insulin varies from 0 to 848. We normalize data to give equal weightage to both the columns
| x = (x_data – np.min(x_data)) / (np.max(x_data) – np.min(x_data)).values |
Upon normalization the data is converted to the range of 0 – 1.
After this data is split into training and testing datasets
| from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size = 0.25, random_state = 42) x_train = x_train.T |
We do this using the built-in method inside the sklearn library.
Defining Necessary Functions
Initialize the weights and biases
The weights and biases are called the parameter of the model. Each feature in the training dataset is given a certain weight. Let’s start by assigning them all some random value i.e. 0.01.
| def initialize_weights_and_bias(dimension):
w = np.full((dimension,1),0.01) |
Sigmoid function
Logistic regression ,uses regression to predict the label by making a linear decision boundary. It would make no sense to get the value greater than 1 or less than 0. Here the sigmoid function comes into play. This function ranges between 0 to 1 and is stated as
| def sigmoid(z):
y_head = 1 / (1+np.exp(-z)) return y_head |
To get the value of the parameter z used in the sigmoid function we use the formula where x is the features array, w weights and b is the bias. Now to make our model learn we need to punish it for the losses and penalize it for wrong predictions this is done by the loss function which is stated as
And cost function is effectively the sum of loss across all the cases.
This process of penalizing the model and moving forward with the predictions is known as forward propagation.
If we recall we allotted random weights to the parameter, they now will be updated based upon our loss function and cost function. To minimize loss and cost function we use gradient descent which is
Where w denotes the weights, denotes the stepsize or the factor by which to change the gradient to find the local minima which are multiplied by the derivative of the loss function to sum it all up the algorithm works as follow:-
We assume a random datapoint in our graph and calculate its slope, then we find the direction in which the loss function decreases and update the weights using the gradient descent formula this process is known as back propagation, we then select point by taking a stepsize of and repeat this entire process once again
sometimes is also referred to as learning rate.
Finally we write the code for forward and backward propagation combined as backward propagation also uses the same z which is found in forward propagation
| def forward_backward_propagation(w,b,x_train,y_head):
z = np.dot(w.T,x_train) + b #backward propogation |
After we have calculated our parameters we need to update the randomly assigned weights to do this we use another update function
| def update(w, b, x_train, y_train, learning_rate,number_of_iterarion): cost_list = [] cost_list2 = [] index = [] # updating(learning) parameters is number_of_iterarion times for i in range(number_of_iterarion): # make forward and backward propagation and find cost and gradients cost,gradients = forward_backward_propagation(w,b,x_train,y_train) cost_list.append(cost) # lets update w = w – learning_rate * gradients[“derivative_weight”] b = b – learning_rate * gradients[“derivative_bias”] if i % 10 == 0: cost_list2.append(cost) index.append(i) print (“Cost after iteration %i: %f” %(i, cost)) parameters = {“weight”: w,”bias”: b}return parameters, gradients, cost_list |
After this function is called our model has successfully calculated the values of weights and biases using the method of forward propagation and backward propagation. This process is also used in neural network i.e. they repeatedly update weights using forward and backward propagation.
Now onto the most awaited part of generating the predictions we do that by defining a predict function as follow
| def predict(w,b,x_test): # x_test is a input for forward propagation z = sigmoid(np.dot(w.T,x_test)+b) Y_prediction = np.zeros((1,x_test.shape[1])) # if z is bigger than 0.5, our prediction is one means has diabete (y_head=1), # if z is smaller than 0.5, our prediction is zero means does not have diabete (y_head=0), for i in range(z.shape[1]): if z[0,i]<= 0.5: Y_prediction[0,i] = 0 else: Y_prediction[0,i] = 1return Y_prediction |
Combining all the functions
| def logistic_regression(x_train, y_train, x_test, y_test, learning_rate , num_iterations): # initialize dimension = x_train.shape[0] w,b = initialize_weights_and_bias(dimension)parameters, gradients, cost_list = update(w, b, x_train, y_train, learning_rate,num_iterations) y_prediction_test = predict(parameters[“weight”],parameters[“bias”],x_test) # Print train/test Errors |
| from sklearn.linear_model import LogisticRegression lr = LogisticRegression() lr.fit(x_train.T,y_train.T) print(“Test Accuracy {}”.format(lr.score(x_test.T,y_test.T))) |
That was all about Logistic Regression without inbuilt python libraries (sklearn).
