Numerical Optimization
Numerical optimization develops algorithms for finding minima or maxima of functions when closed-form solutions are unavailable. It balances convergence, stability, and computational cost.
Key Points
- Line search methods ensure sufficient decrease at each step.
- Trust-region methods model the objective locally and restrict step size.
- Stochastic and distributed variants scale optimization to massive datasets.
Formulas
Wolfe conditions
$$f(x_k + \alpha p_k) \le f(x_k) + c_1 \alpha \nabla f_k^\top p_k, \quad \nabla f(x_k + \alpha p_k)^\top p_k \ge c_2 \nabla f_k^\top p_k$$
Trust-region subproblem
$$\min_{\|p\| \le \Delta} m_k(p)$$
Code Example
from scipy.optimize import minimize
res = minimize(lambda x: x[0]**2 + 10*x[1]**2, [1.0, 1.0], method='BFGS')
print(res.x) # ~ [0, 0]