Machine Learning / DL / LLM Difficulty: Intermediate

Linear Regression

Linear regression models the relationship between a dependent variable and one or more independent variables via a linear function. The least-squares estimator has a closed-form solution and rich statistical properties.

Key Points

  • The ordinary least squares (OLS) estimator minimizes the residual sum of squares.
  • Under the Gauss-Markov assumptions, OLS is the best linear unbiased estimator (BLUE).
  • Regularization (Ridge, Lasso) trades bias for variance and improves generalization.

Formulas

OLS estimator
$$\hat{\beta} = (X^\top X)^{-1} X^\top y$$
Ridge regression
$$\hat{\beta}_{ridge} = \arg\min_\beta \|y - X\beta\|^2 + \lambda \|\beta\|^2$$
Prediction
$$\hat{y} = X \hat{\beta}$$

Code Example

import numpy as np

X = np.random.randn(100, 3)
y = X @ np.array([1, -2, 3]) + 0.1*np.random.randn(100)
beta = np.linalg.lstsq(X, y, rcond=None)[0]
print(beta)

Tags

  • supervised-learning
  • least-squares
  • regression

References

  • The Elements of Statistical Learning
    Trevor Hastie, Robert Tibshirani, and Jerome Friedman · Springer · source
  • Pattern Recognition and Machine Learning
    Christopher M. Bishop · Springer · source

Knowledge Graph