Machine Learning / DL / LLM Difficulty: Intermediate

Logistic Regression

Logistic regression models binary probabilities using the logistic function. It is a convex classification method and a building block for neural networks.

Key Points

  • The logistic (sigmoid) function maps real values to (0,1).
  • Maximum likelihood estimation yields a convex optimization problem.
  • Newton's method (iteratively reweighted least squares) solves it efficiently.

Formulas

Sigmoid
$$\sigma(z) = \frac{1}{1 + e^{-z}}$$
Log-odds
$$\log \frac{p}{1-p} = X\beta$$
Log-likelihood
$$\ell(\beta) = \sum_{i=1}^n y_i X_i^\top \beta - \log(1 + e^{X_i^\top \beta})$$

Code Example

from sklearn.linear_model import LogisticRegression

# X: features, y: binary labels
clf = LogisticRegression().fit(X, y)
print(clf.coef_)

Tags

  • classification
  • generalized-linear-models

References

  • The Elements of Statistical Learning
    Trevor Hastie, Robert Tibshirani, and Jerome Friedman · Springer · source
  • Pattern Recognition and Machine Learning
    Christopher M. Bishop · Springer · source

Knowledge Graph