Logistic regression gradient descent in Python

What gradient descent does

A logistic model first calculates a score, often written z = b + w₁x₁ + …. A sigmoid function changes that score into a probability from 0 to 1. The model compares those probabilities with the known 0/1 answers using log loss.

Gradient descent calculates how each weight affects that loss. It then changes the weights a little. If the learning rate is well chosen, repeated updates make the training loss smaller. The goal is not merely to make the training loss tiny; the model also needs to work on new cases.

The update rule in plain English

For a data matrix X, predicted probabilities p, target labels y, and n rows, the full-batch gradient for logistic loss is:

gradient = Xᵀ(p − y) / n

Then the program applies new weights = old weights − learning rate × gradient. The minus sign moves the weights downhill on the loss curve. This simple form assumes binary labels and the usual logistic loss; regularization adds another term.

Worked Python example

The input values are scaled practice scores. The labels deliberately overlap: the same range contains both 0 and 1 outcomes, so a finite logistic fit is possible without perfectly separating the classes.

import numpy as np

# One scaled feature and a binary target.
x = np.array([-1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0])
y = np.array([0, 0, 1, 0, 1, 1, 0, 1], dtype=float)

# First column is the intercept; second column is the feature.
X = np.column_stack([np.ones(len(x)), x])
weights = np.zeros(X.shape[1])
learning_rate = 0.1
loss_history = []

def sigmoid(z):
    z = np.clip(z, -30, 30)  # avoid overflow in exp()
    return 1 / (1 + np.exp(-z))

def record_loss(probability):
    probability = np.clip(probability, 1e-12, 1 - 1e-12)
    return -np.mean(y * np.log(probability) + (1 - y) * np.log(1 - probability))

loss_history.append(record_loss(sigmoid(X @ weights)))  # loss before updates
for step in range(1, 3001):
    probability = sigmoid(X @ weights)
    gradient = X.T @ (probability - y) / len(y)
    weights -= learning_rate * gradient
    if step % 100 == 0:
        loss_history.append(record_loss(sigmoid(X @ weights)))

new_x = np.array([-0.25, 0.75])
new_X = np.column_stack([np.ones(len(new_x)), new_x])
new_probability = sigmoid(new_X @ weights)

print("intercept, slope:", weights)
print("probability of class 1:", new_probability)
print("final training log loss:", loss_history[-1])

The first weight is the intercept and the second is the slope for the scaled input. The final numbers are probabilities, not certain answers. The chart above records the loss every 100 updates from this exact example.

For a production model, use a train/validation split, inspect class balance and compare suitable metrics. The simple loop omits regularization and stopping rules to make the update visible.

Common problems and how to spot them

• Loss jumps or becomes NaN: the learning rate may be too large, or the exponential calculation may overflow. Scale features and use a numerically safe sigmoid.
• Loss barely changes: the learning rate may be too small, or the data may not contain a useful signal.
• Training looks good but validation is poor: the model may overfit, the features may be weak, or the validation set may differ from training data.
• One class is rare: accuracy alone can look high while the model misses most rare cases. Check recall, precision, the confusion matrix, and probability quality.

scikit-learn’s SGDClassifier(loss="log_loss") fits logistic regression with stochastic gradient descent. It updates from individual samples rather than calculating one full-data gradient at each step, so its path and final values can differ from this teaching loop.

Frequently asked questions

Sources and next steps

• scikit-learn: Stochastic Gradient Descent — SGD behavior, feature scaling and logistic loss.
• scikit-learn: SGDClassifier — the log_loss setting for logistic classification.
• scikit-learn: Logistic regression — probability and loss definitions.