|
|
| Line 1: |
Line 1: |
| In [[Optimization (mathematics)|optimization]], '''quasi-Newton methods''' (a special case of '''variable metric methods''') are algorithms for finding local [[maxima and minima]] of [[function (mathematics)|functions]]. Quasi-Newton methods are based on
| | Hi there, I am Alyson Boon even though it is not the title on my birth certification. Office supervising is my profession. Ohio is where my house is but my husband desires us to transfer. What me and my family adore is to climb but I'm considering on beginning some thing new.<br><br>My website :: [http://www.khuplaza.com/dent/14869889 best psychic] |
| [[Newton's method in optimization|Newton's method]] to find the [[stationary point]] of a function, where the [[gradient]] is 0. Newton's method assumes that the function can be locally approximated as a [[quadratic function|quadratic]] in the region around the optimum, and uses the first and second derivatives to find the stationary point. In higher dimensions, Newton's method uses the gradient and the [[Hessian matrix]] of second [[derivative]]s of the function to be minimized.
| |
|
| |
| In quasi-Newton methods the Hessian matrix does not need to be computed. The Hessian is updated by analyzing successive gradient vectors instead. Quasi-Newton methods are a generalization of the [[secant method]] to find the root of the first derivative for multidimensional problems. In multiple dimensions the secant equation is under-determined, and quasi-Newton methods differ in how they constrain the solution, typically by adding a simple low-rank update to the current estimate of the Hessian.
| |
| | |
| The first quasi-Newton algorithm was proposed by [[William C. Davidon]], a physicist working at [[Argonne National Laboratory]]. He developed the first quasi-Newton algorithm in 1959: the [[DFP updating formula]], which was later popularized by Fletcher and Powell in 1963, but is rarely used today. The most common quasi-Newton algorithms are currently the [[SR1 formula]] (for symmetric rank one), the [[BHHH]] method, the widespread [[BFGS method]] (suggested independently by Broyden, Fletcher, Goldfarb, and Shanno, in 1970), and its low-memory extension, [[L-BFGS]]. The Broyden's class is a linear combination of the DFP and BFGS methods.
| |
| | |
| The SR1 formula does not guarantee the update matrix to maintain [[Positive-definite matrix|positive-definiteness]] and can be used for indefinite problems.
| |
| The [[Broyden's method]] does not require the update matrix to be symmetric and it is used to find the root of a general system of equations (rather than the gradient)
| |
| by updating the [[Jacobian matrix and determinant|Jacobian]] (rather than the Hessian).
| |
| | |
| One of the chief advantages of quasi-Newton methods over [[Newton's method in optimization|Newton's method]] is that the [[Hessian matrix]] (or, in the case of quasi-Newton methods, its approximation) <math>B</math> does not need to be inverted. Newton's method, and its derivatives such as [[interior point method]]s, require the Hessian to be inverted, which is typically implemented by solving a [[system of linear equations]] and is often quite costly. In contrast, quasi-Newton methods usually generate an estimate of <math>B^{-1}</math> directly.
| |
| | |
| ==Description of the method==
| |
| As in [[Newton's method in optimization|Newton's method]], one uses a second order approximation to find the minimum of a function <math>f(x)</math>.
| |
| The [[Taylor series]] of <math>f(x)</math> around an iterate is:
| |
| ::<math>f(x_k+\Delta x) \approx f(x_k)+\nabla f(x_k)^T \Delta x+\frac{1}{2} \Delta x^T {B} \, \Delta x, </math>
| |
| where (<math>\nabla f</math>) is the [[gradient]] and <math>B</math> an approximation to the [[Hessian matrix]]. | |
| The gradient of this approximation (with respect to <math> \Delta x </math>) is
| |
| | |
| ::<math> \nabla f(x_k+\Delta x) \approx \nabla f(x_k)+B \, \Delta x</math>
| |
| | |
| and setting this gradient to zero provides the Newton step: | |
| | |
| ::<math>\Delta x=-B^{-1}\nabla f(x_k), \, </math>
| |
| | |
| The Hessian approximation <math> B </math> is chosen to satisfy
| |
| | |
| ::<math>\nabla f(x_k+\Delta x)=\nabla f(x_k)+B \, \Delta x,</math>
| |
| | |
| which is called the ''secant equation'' (the Taylor series of the gradient itself). In more than one dimension <math>B</math> is [[under determined]]. In one dimension, solving for <math>B</math> and applying the Newton's step with the updated value is equivalent to the [[secant method]].
| |
| The various quasi-Newton methods differ in their choice of the solution to the secant equation (in one dimension, all the variants are equivalent).
| |
| Most methods (but with exceptions, such as [[Broyden's method]]) seek a symmetric solution (<math>B^T=B</math>); furthermore,
| |
| the variants listed below can be motivated by finding an update <math>B_{k+1}</math> that is as close as possible
| |
| to <math> B_{k}</math> in some [[Norm (mathematics)|norm]]; that is, <math> B_{k+1} = \textrm{argmin}_B \|B-B_k\|_V </math> where <math>V </math> is some [[positive definite matrix]] that defines the norm.
| |
| An approximate initial value of <math>B_0=I * x</math> is often sufficient to achieve rapid convergence. The unknown <math>x_k</math> is updated applying the Newton's step calculated using the current approximate Hessian matrix <math>B_{k}</math>
| |
| * <math>\Delta x_k=- \alpha_k B_k^{-1}\nabla f(x_k)</math>, with <math>\alpha</math> chosen to satisfy the [[Wolfe conditions]];
| |
| * <math>x_{k+1}=x_{k}+\Delta x_k</math>;
| |
| *The gradient computed at the new point <math>\nabla f(x_{k+1})</math>, and
| |
| :<math>y_k=\nabla f(x_{k+1})-\nabla f(x_k),</math>
| |
| is used to update the approximate Hessian <math>\displaystyle B_{k+1}</math>, or directly its inverse <math>\displaystyle H_{k+1}=B_{k+1}^{-1}</math> using the [[Sherman-Morrison formula]].
| |
| * A key property of the BFGS and DFP updates is that if <math> B_k </math> is positive definite and <math> \alpha_k </math> is chosen to satisfy the Wolfe conditions then <math>\displaystyle B_{k+1} </math> is also positive definite.
| |
| | |
| The most popular update formulas are:
| |
| | |
| {| class="wikitable"
| |
| | |
| |-
| |
| ! Method
| |
| ! <math>\displaystyle B_{k+1}=</math>
| |
| ! <math>H_{k+1}=B_{k+1}^{-1}=</math>
| |
| |-
| |
| | |
| | [[DFP updating formula|DFP]]
| |
| | |
| | <math>\left (I-\frac {y_k \, \Delta x_k^T} {y_k^T \, \Delta x_k} \right ) B_k \left (I-\frac {\Delta x_k y_k^T} {y_k^T \, \Delta x_k} \right )+\frac{y_k y_k^T} {y_k^T \, \Delta x_k}</math>
| |
| | |
| | <math> H_k + \frac {\Delta x_k \Delta x_k^T}{y_k^{T} \, \Delta x_k} - \frac {H_k y_k y_k^T H_k^T} {y_k^T H_k y_k}</math>
| |
| | |
| |-
| |
| | |
| | [[BFGS method|BFGS]]
| |
| | |
| | <math> B_k + \frac {y_k y_k^T}{y_k^{T} \Delta x_k} - \frac {B_k \Delta x_k (B_k \Delta x_k)^T} {\Delta x_k^{T} B_k \, \Delta x_k}</math>
| |
| | |
| | <math> \left (I-\frac {y_k \Delta x_k^T} {y_k^T \Delta x_k} \right )^T H_k \left (I-\frac { y_k \Delta x_k^T} {y_k^T \Delta x_k} \right )+\frac
| |
| {\Delta x_k \Delta x_k^T} {y_k^T \, \Delta x_k}</math>
| |
| | |
| |-
| |
| | [[Broyden's method|Broyden]]
| |
| | |
| | <math> B_k+\frac {y_k-B_k \Delta x_k}{\Delta x_k^T \, \Delta x_k} \, \Delta x_k^T </math>
| |
| | |
| |<math>H_{k}+\frac {(\Delta x_k-H_k y_k) \Delta x_k^T H_k}{\Delta x_k^T H_k \, y_k}</math>
| |
| | |
| |-
| |
| | |
| | Broyden family
| |
| | |
| | <math>(1-\varphi_k) B_{k+1}^{BFGS}+ \varphi_k B_{k+1}^{DFP}, \qquad \varphi\in[0,1]</math>
| |
| | |
| |
| |
| |-
| |
| | |
| | [[SR1 formula|SR1]]
| |
| | |
| | <math>B_{k}+\frac {(y_k-B_k \, \Delta x_k) (y_k-B_k \, \Delta x_k)^T}{(y_k-B_k \, \Delta x_k)^T \, \Delta x_k}</math>
| |
| | |
| | <math>H_{k}+\frac {(\Delta x_k-H_k y_k) (\Delta x_k-H_k y_k)^T}{(\Delta x_k-H_k y_k)^T y_k}</math>
| |
| | |
| |}
| |
| | |
| ==Implementations==
| |
| Owing to their success, there are implementations of quasi-Newton methods in almost all programming languages. The [[NAG Numerical Library|NAG Library]] contains several routines<ref>{{ cite web | last = The Numerical Algorithms Group | first = | title = Keyword Index: Quasi-Newton | date = | work = NAG Library Manual, Mark 23 | url = http://www.nag.co.uk/numeric/fl/nagdoc_fl23/html/INDEXES/KWIC/quasi-newton.html | accessdate = 2012-02-09 }}</ref> for minimizing or maximizing a function<ref>{{ cite web | last = The Numerical Algorithms Group | first = | title = E04 – Minimizing or Maximizing a Function | date = | work = NAG Library Manual, Mark 23 | url = http://www.nag.co.uk/numeric/fl/nagdoc_fl23/pdf/E04/e04intro.pdf | accessdate = 2012-02-09 }}</ref> which use quasi-Newton algorithms.
| |
| | |
| In MATLAB's [[Optimization Toolbox]], the <code>[http://www.mathworks.com/help/toolbox/optim/ug/fminunc.html fminunc]</code> function uses (among other methods) the [[BFGS]] Quasi-Newton method. Many of the [http://www.mathworks.com/help/toolbox/optim/ug/brnoxzl.html constrained methods] of the Optimization toolbox use [[BFGS]] and the variant [[L-BFGS]]. Many user-contributed quasi-Newton routines are available on MATLAB's [http://www.mathworks.com/matlabcentral/fileexchange/?term=BFGS file exchange].
| |
| | |
| [[Mathematica]] includes [http://reference.wolfram.com/mathematica/tutorial/UnconstrainedOptimizationQuasiNewtonMethods.html quasi-Newton solvers]. [[R (programming language)|R]]'s <code>optim</code> general-purpose optimizer routine uses the [[BFGS]] method by using <code>method="BFGS"</code><cite>[http://finzi.psych.upenn.edu/R/library/stats/html/optim.html]</cite>. In the [[SciPy]] extension to [[Python (programming language)|Python]], the [http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html <code>scipy.optimize.minimize</code>] function includes, among other methods, a [[BFGS]] implementation.
| |
| | |
| ==See also==
| |
| * [[Newton's method in optimization]]
| |
| * [[Newton's method]]
| |
| * [[DFP updating formula]]
| |
| * [[BFGS method]]
| |
| :*[[Limited-memory BFGS|L-BFGS]]
| |
| :*[[Orthant-wise limited-memory quasi-Newton|OWL-QN]]
| |
| * [[SR1 formula]]
| |
| * [[Broyden's Method]]
| |
| | |
| == References ==
| |
| | |
| {{reflist}}
| |
| | |
| == Further reading ==
| |
| * Bonnans, J. F., Gilbert, J.Ch., [[Claude Lemaréchal|Lemaréchal, C.]] and Sagastizábal, C.A. (2006), ''Numerical optimization, theoretical and numerical aspects.'' Second edition. Springer. ISBN 978-3-540-35445-1.
| |
| * William C. Davidon, [http://link.aip.org/link/?SJE/1/1/1 Variable Metric Method for Minimization], SIOPT Volume 1 Issue 1, Pages 1–17, 1991.
| |
| * {{Citation | last1=Fletcher | first1=Roger | title=Practical methods of optimization | publisher=[[John Wiley & Sons]] | location=New York | edition=2nd | isbn=978-0-471-91547-8 | year=1987}}.
| |
| * Nocedal, Jorge & Wright, Stephen J. (1999). Numerical Optimization. Springer-Verlag. ISBN 0-387-98793-2.
| |
| *{{Cite book | last1=Press | first1=WH | last2=Teukolsky | first2=SA | last3=Vetterling | first3=WT | last4=Flannery | first4=BP | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press | publication-place=New York | isbn=978-0-521-88068-8 | chapter=Section 10.9. Quasi-Newton or Variable Metric Methods in Multidimensions | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=521}}
| |
| <!-- * Edwin K.P.Chong and Stanislaw H.Zak, An Introduction to Optimization 2ed, John Wiley & Sons Pte. Ltd. August 2001. -->
| |
| | |
| {{Optimization algorithms|unconstrained}}
| |
| | |
| [[Category:Optimization algorithms and methods]]
| |
Hi there, I am Alyson Boon even though it is not the title on my birth certification. Office supervising is my profession. Ohio is where my house is but my husband desires us to transfer. What me and my family adore is to climb but I'm considering on beginning some thing new.
My website :: best psychic