1. **Classical Optimization using Differential Calculus (Single Variable):** Optimization involves finding the maximum or minimum values of a function. For a single variable function $f(x)$, critical points occur where the derivative $f'(x) = 0$ or is undefined. **Steps:** - Find $f'(x)$. - Solve $f'(x) = 0$ to find critical points. - Use the second derivative test: if $f''(x) > 0$, local minimum; if $f''(x) < 0$, local maximum. 2. **Multivariable Optimization without Constraints:** For a function $f(x,y)$, find points where the gradient vector $\nabla f = (f_x, f_y) = (0,0)$. **Steps:** - Compute partial derivatives $f_x$ and $f_y$. - Solve $f_x=0$ and $f_y=0$ simultaneously. - Use the Hessian matrix $H = \begin{bmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{bmatrix}$. - Determine definiteness of $H$ at critical points: positive definite for minima, negative definite for maxima, indefinite for saddle points. 3. **Multivariable Optimization with Constraints (Lagrangian Theory):** To optimize $f(x,y,...)$ subject to constraint(s) $g(x,y,...) = 0$, use Lagrange multipliers. **Steps:** - Form Lagrangian $\mathcal{L} = f - \lambda g$. - Compute partial derivatives of $\mathcal{L}$ with respect to variables and $\lambda$. - Solve the system $\nabla f = \lambda \nabla g$ and $g=0$. 4. **Kuhn-Tucker Conditions (for inequality constraints):** For problems with inequality constraints $g_i(x) \leq 0$, the Kuhn-Tucker conditions generalize Lagrange multipliers. **Conditions:** - Stationarity: $\nabla f + \sum \mu_i \nabla g_i = 0$. - Primal feasibility: $g_i(x) \leq 0$. - Dual feasibility: $\mu_i \geq 0$. - Complementary slackness: $\mu_i g_i(x) = 0$. --- **Example 1: Single Variable Optimization** Maximize $f(x) = -2x^2 + 4x + 1$. 1. Find derivative: $f'(x) = -4x + 4$. 2. Set $f'(x) = 0 \Rightarrow -4x + 4 = 0 \Rightarrow x = 1$. 3. Second derivative: $f''(x) = -4 < 0$, so $x=1$ is a maximum. 4. Maximum value: $f(1) = -2(1)^2 + 4(1) + 1 = 3$. **Answer:** Maximum at $x=1$ with value 3. --- **Example 2: Multivariable Optimization without Constraints** Find extrema of $f(x,y) = x^2 + y^2 - 4x - 6y + 13$. 1. Compute partial derivatives: $$f_x = 2x - 4, \quad f_y = 2y - 6$$ 2. Set to zero: $$2x - 4 = 0 \Rightarrow x=2$$ $$2y - 6 = 0 \Rightarrow y=3$$ 3. Hessian matrix: $$H = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$$ Positive definite since eigenvalues are positive. 4. So $(2,3)$ is a minimum. 5. Minimum value: $$f(2,3) = 4 + 9 - 8 - 18 + 13 = 0$$ **Answer:** Minimum at $(2,3)$ with value 0. --- **Example 3: Multivariable Optimization with Constraint (Lagrangian)** Maximize $f(x,y) = xy$ subject to $x^2 + y^2 = 1$. 1. Form Lagrangian: $$\mathcal{L} = xy - \lambda (x^2 + y^2 - 1)$$ 2. Compute partial derivatives: $$\frac{\partial \mathcal{L}}{\partial x} = y - 2\lambda x = 0$$ $$\frac{\partial \mathcal{L}}{\partial y} = x - 2\lambda y = 0$$ $$x^2 + y^2 = 1$$ 3. From first two equations: $$y = 2\lambda x, \quad x = 2\lambda y$$ 4. Substitute $y$ into second: $$x = 2\lambda (2\lambda x) = 4\lambda^2 x$$ 5. If $x \neq 0$, then $1 = 4\lambda^2 \Rightarrow \lambda = \pm \frac{1}{2}$. 6. For $\lambda = \frac{1}{2}$, $y = x$; for $\lambda = -\frac{1}{2}$, $y = -x$. 7. Using constraint: $$x^2 + y^2 = 1 \Rightarrow 2x^2 = 1 \Rightarrow x = \pm \frac{1}{\sqrt{2}}$$ 8. Points: $\left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right), \left(-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}\right), \left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}\right), \left(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)$ 9. Evaluate $f$: $$f = xy = \pm \frac{1}{2}$$ **Answer:** Maximum $\frac{1}{2}$ at $(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$ and $(-\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})$; minimum $-\frac{1}{2}$ at $(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})$ and $(-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$. --- **Example 4: Kuhn-Tucker Conditions** Maximize $f(x) = x$ subject to $x^2 \leq 1$. 1. Constraint: $g(x) = x^2 - 1 \leq 0$. 2. Stationarity: $$f'(x) + \mu g'(x) = 1 + \mu (2x) = 0$$ 3. Primal feasibility: $x^2 - 1 \leq 0$. 4. Dual feasibility: $\mu \geq 0$. 5. Complementary slackness: $\mu (x^2 - 1) = 0$. Check cases: - If $|x| < 1$, then $g(x) < 0$, so $\mu = 0$. - Then $1 + 0 = 1 \neq 0$, no solution. - If $|x| = 1$, then $g(x) = 0$, complementary slackness allows $\mu \geq 0$. - Stationarity: $1 + 2\mu x = 0 \Rightarrow \mu = -\frac{1}{2x}$. - For $x=1$, $\mu = -\frac{1}{2} < 0$ invalid. - For $x=-1$, $\mu = \frac{1}{2} \geq 0$ valid. **Answer:** Maximum at $x = -1$ with $f(-1) = -1$ under Kuhn-Tucker conditions.