|
|
| Line 1: |
Line 1: |
| In [[mathematics]], '''Laplace's method''', named after [[Pierre-Simon Laplace]], is a technique used to approximate [[integral]]s of the form
| | I am Margarita from Sjallands Odde doing my final year engineering in Latin American Studies. I did my schooling, secured 86% and hope to find someone with same interests in Games Club - Dungeons and Dragons, Monopoly, Etc..<br><br>my site: [http://www.bayviewgroup.com.au/ForumRetrieve.aspx?ForumID=317&TopicID=581177&NoTemplate=False Bookbyte Free Shipping Code] |
| | |
| :<math> \int_a^b\! e^{M f(x)} \, dx </math>
| |
| | |
| where ''ƒ''(''x'') is some twice-[[Derivative|differentiable]] [[function (mathematics)|function]], ''M'' is a large number, and the integral endpoints ''a'' and ''b'' could possibly be infinite. This technique was originally presented in Laplace (1774, pp. 366–367).
| |
| | |
| == The idea of Laplace's method ==
| |
| [[File:Laplaces method.svg|right|150px|thumb|The function ''e''<sup>''Mƒ''(''x'')</sup>, in blue, is shown on top for ''M'' = 0.5, and at the bottom for ''M'' = 3. Here, ''ƒ''(''x'') = sin ''x''/''x'', with a global maximum at ''x''<sub>0</sub> = 0. It is seen that as ''M'' grows larger, the approximation of this function by a [[Gaussian function]] (shown in red) is getting better. This observation underlies Laplace's method.]]
| |
| | |
| Assume that the function ''ƒ''(''x'') has a unique [[Maxima and minima|global maximum]] at ''x''<sub>0</sub>. Then, the value ''ƒ''(''x''<sub>0</sub>) will be larger than other values ''ƒ''(''x''). If we multiply this function by a large number ''M'', the ratio between ''Mƒ''(''x''<sub>0</sub>) and ''Mƒ''(''x'') will stay the same (since ''Mƒ''(''x''<sub>0</sub>)/''Mƒ''(''x'') = ''ƒ''(''x''<sub>0</sub>)/''ƒ''(''x'')), but it will grow exponentially in the function (see figure)
| |
| | |
| :<math> e^{M f(x)}. \,</math>
| |
| | |
| Thus, significant contributions to the integral of this function will come only from points ''x'' in a [[Topology glossary#N|neighborhood]] of ''x''<sub>0</sub>, which can then be estimated.
| |
| | |
| ==General theory of Laplace's method==
| |
| To state and motivate the method, we need several assumptions. We will assume that ''x''<sub>0</sub> is not an endpoint of the interval of integration, that the values ''ƒ''(''x'') cannot be very close to ''ƒ''(''x''<sub>0</sub>) unless ''x'' is close to ''x''<sub>0</sub>, and that the second derivative <math>f''(x_0)<0</math>.
| |
| | |
| We can expand ''ƒ''(''x'') around ''x''<sub>0</sub> by [[Taylor's theorem]],
| |
| | |
| :<math>f(x) = f(x_0) + f'(x_0)(x-x_0) + \frac{1}{2} f''(x_0)(x-x_0)^2 + R </math>
| |
| | |
| :where <math>R = O\left((x-x_0)^3\right).</math>
| |
| | |
| Since ''ƒ'' has a global maximum at ''x''<sub>0</sub>, and since ''x''<sub>0</sub> is not an endpoint, it is a [[stationary point]], so the derivative of ''ƒ'' vanishes at ''x''<sub>0</sub>. Therefore, the function ''ƒ''(''x'') may be approximated to quadratic order
| |
| | |
| :<math> f(x) \approx f(x_0) - \frac{1}{2} |f''(x_0)| (x-x_0)^2</math>
| |
| | |
| for ''x'' close to ''x''<sub>0</sub> (recall that the second derivative is negative at the global maximum ''ƒ''(''x''<sub>0</sub>)). The assumptions made ensure the accuracy of the approximation
| |
| | |
| :<math>\int_a^b\! e^{M f(x)}\, dx\approx e^{M f(x_0)}\int_a^b e^{-M|f''(x_0)| (x-x_0)^2/2} \, dx</math>
| |
| | |
| (see the picture on the right). This latter integral is a [[Gaussian integral]] if the limits of integration go from −∞ to +∞ (which can be assumed because the exponential decays very fast away from ''x''<sub>0</sub>), and thus it can be calculated. We find
| |
| | |
| :<math>\int_a^b\! e^{M f(x)}\, dx\approx \sqrt{\frac{2\pi}{M|f''(x_0)|}}e^{M f(x_0)} \text { as } M\to\infty. \,</math>
| |
| | |
| A generalization of this method and extension to arbitrary precision is provided by Fog (2008).
| |
| | |
| '''Formal statement and proof:'''
| |
| | |
| Assume that <math> f(x) </math> is a twice differentiable function on <math> [a,b] </math> with <math> x_0 \in [a,b] </math> the unique point such that <math> f(x_0) = \max_{[a,b]} f(x) </math>. Assume additionally that <math> f''(x_0)<0 </math>.
| |
| | |
| Then,
| |
| | |
| : <math>
| |
| \lim_{n \to +\infty} \left( \frac{\int_a^b e^{nf(x)} \, dx}{\left( e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} \right)} \right) =1
| |
| </math>
| |
| | |
| {{cot|'''Proof:'''}}
| |
| '''Lower bound''':
| |
| | |
| Let <math> \varepsilon > 0 </math>. Then by the continuity of <math> f'' </math> there exists <math> \delta >0 </math> such that if <math> |x_0-c|< \delta </math> then <math> f''(c) \ge f''(x_0) - \varepsilon. </math>. By [[Taylor's Theorem]], for any <math> x \in (x_0 - \delta, x_0 + \delta) </math>, <math>f(x) \ge f(x_0) + \frac{1}{2}(f''(x_0) - \varepsilon)(x-x_0)^2 </math>.
| |
| | |
| Then we have the following lower bound:
| |
| | |
| : <math>
| |
| \int_a^b e^{n f(x) } \, dx \ge \int_{x_0 - \delta}^{x_0 + \delta} e^{n f(x)} \, dx
| |
| \ge e^{n f(x_0)} \int_{x_0 - \delta}^{x_0 + \delta} e^{\frac{n}{2} (f''(x_0) - \varepsilon)(x-x_0)^2} \, dx
| |
| = e^{n f(x_0)} \sqrt{\frac{1}{n (-f''(x_0) + \varepsilon)}} \int_{-\delta \sqrt{n (-f''(x_0) + \varepsilon)} }^{\delta \sqrt{n (-f''(x_0) + \varepsilon)} } e^{-\frac{1}{2}y^2} \, dy
| |
| </math>
| |
| | |
| where the last equality was obtained by a change of variables <math> y= \sqrt{n (-f''(x_0) + \varepsilon)} (x-x_0)</math>. Remember that <math> f''(x_0)<0 </math> so that is why we can take the square root of its negation.
| |
| | |
| If we divide both sides of the above inequality by <math> e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} </math> and take the limit we get:
| |
| | |
| : <math>
| |
| \lim_{n \to +\infty} \left( \frac{\int_a^b e^{nf(x)} \,dx}{\left( e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} \right)} \right)
| |
| \ge \lim_{n \to +\infty} \frac{1}{\sqrt{2 \pi}} \int_{-\delta\sqrt{n (-f''(x_0) + \varepsilon)} }^{\delta \sqrt{n (-f''(x_0) + \varepsilon)} } e^{-\frac{1}{2}y^2} \, dy \sqrt{\frac{-f''(x_0)}{-f''(x_0) + \varepsilon}}
| |
| = \sqrt{\frac{-f''(x_0)}{-f''(x_0) + \varepsilon}}
| |
| </math>
| |
| | |
| since this is true for arbitrary <math> \varepsilon </math> we get the lower bound:
| |
| | |
| : <math>
| |
| \lim_{n \to +\infty} \left( \frac{\int_a^b e^{nf(x)} \, dx}{\left( e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} \right)} \right) \ge 1
| |
| </math>
| |
| | |
| Note that this proof works also when <math> a = -\infty </math> or <math> b= \infty </math> (or both).
| |
| | |
| '''Upper bound''':
| |
| | |
| The proof of the upper bound is similar to the proof of the lower bound but there are a few inconveniences. Again we start by picking an <math> \varepsilon >0 </math> but in order for the proof to work we need <math> \varepsilon </math> small enough so that <math> f''(x_0) + \varepsilon < 0 </math>. Then, as above, by continuity of <math> f'' </math> and [[Taylor's Theorem]] we can find <math> \delta>0 </math> so that if <math> |x-x_0| < \delta </math>, then <math> f(x) \le f(x_0) + \frac{1}{2} (f''(x_0) + \varepsilon)(x-x_0)^2 </math>. Lastly, by our assumptions (assuming <math> a,b </math> are finite) there exists an <math> \eta >0 </math> such that if <math> |x-x_0|\ge \delta </math>, then <math> f(x) \le f(x_0) - \eta </math>.
| |
| | |
| Then we can calculate the following upper bound:
| |
| | |
| : <math>
| |
| \int_a^b e^{n f(x) } \, dx
| |
| \le \int_a^{x_0-\delta} e^{n f(x) } \, dx + \int_{x_0-\delta}^{x_0 + \delta} e^{n f(x) } \, dx + \int_{x_0 + \delta}^b e^{n f(x) } \, dx
| |
| \le (b-a)e^{n (f(x_0) - \eta)} + \int_{x_0-\delta}^{x_0 + \delta} e^{n f(x) } \, dx
| |
| </math>
| |
| | |
| : <math>
| |
| \le (b-a)e^{n (f(x_0) - \eta)} + e^{n f(x_0)} \int_{x_0-\delta}^{x_0 + \delta} e^{\frac{n}{2} (f''(x_0)+\varepsilon)(x-x_0)^2} \, dx
| |
| \le (b-a)e^{n (f(x_0) - \eta)} + e^{n f(x_0)} \int_{-\infty}^{+\infty} e^{\frac{n}{2} (f''(x_0)+\varepsilon)(x-x_0)^2} \, dx
| |
| </math>
| |
| | |
| : <math>
| |
| \le (b-a)e^{n (f(x_0) - \eta)} + e^{n f(x_0)} \sqrt{\frac{2 \pi}{n (-f''(x_0) - \varepsilon)}}
| |
| </math>
| |
| | |
| If we divide both sides of the above inequality by <math> e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} </math> and take the limit we get:
| |
| | |
| : <math>
| |
| \lim_{n \to +\infty} \left( \frac{\int_a^b e^{nf(x)} \, dx}{\left( e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} \right)} \right)
| |
| \le \lim_{n \to +\infty} \left( (b-a) e^{-\eta n} \sqrt{\frac{n (-f''(x_0))}{2 \pi}} + \sqrt{\frac{-f''(x_0)}{-f''(x_0) - \varepsilon}} \right)
| |
| = \sqrt{\frac{-f''(x_0)}{-f''(x_0) - \varepsilon}}
| |
| </math>
| |
| | |
| Since <math> \varepsilon </math> is arbitrary we get the upper bound:
| |
| | |
| : <math>
| |
| \lim_{n \to +\infty} \left( \frac{\int_a^b e^{nf(x)} \, dx}{\left( e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} \right)} \right) \le 1
| |
| </math>
| |
| | |
| And combining this with the lower bound gives the result.
| |
| | |
| Note that the above proof obviously fails when <math> a = -\infty </math> or <math> b = \infty </math> (or both). To deal with these cases, we need some extra assumptions. A sufficient (not necessary) assumption is that for <math> n = 1 </math>, the integral <math> \int_a^b e^{nf(x)} \, dx </math> is finite, and that the number <math> \eta </math> as above exists (note that this must be an assumption in the case when the interval <math> [a,b] </math> is infinite). The proof proceeds otherwise as above, but the integrals
| |
| | |
| : <math>
| |
| \int_a^{x_0-\delta} e^{n f(x) } \, dx + \int_{x_0 + \delta}^b e^{n f(x) } \, dx
| |
| </math>
| |
| | |
| must be approximated by
| |
| | |
| : <math>
| |
| \int_a^{x_0-\delta} e^{n f(x) } \, dx + \int_{x_0 + \delta}^b e^{n f(x) } \, dx
| |
| \le \int_a^b e^{f(x)}e^{(n-1)(f(x_0) - \eta)} \, dx = e^{(n-1)(f(x_0) - \eta)} \int_a^b e^{f(x)} \, dx
| |
| </math>
| |
| | |
| instead of <math> (b-a)e^{n (f(x_0) - \eta)} </math> as above, so that when we divide by <math> e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}} </math>, we get for this term
| |
| | |
| : <math>
| |
| \frac{e^{(n-1)(f(x_0) - \eta)} \int_a^b e^{f(x)} \, dx }{e^{nf(x_0)}\sqrt{\frac{2 \pi}{n (-f''(x_0))}}} = e^{-(n-1)\eta} \sqrt{n} e^{-f(x_0)} \int_a^b e^{f(x)} \, dx \sqrt{\frac{ -f''(x_0)}{ 2 \pi}}
| |
| </math>
| |
| | |
| whose limit as <math> n \rightarrow \infty </math> is <math> 0 </math>. The rest of the proof (the analysis of the interesting term) proceeds as above.
| |
| | |
| The given condition in the infinite interval case is, as said above, sufficient but not necessary. However, the condition is fulfilled in many, if not in most, applications: the condition simply says that the integral we are studying must be well-defined (not infinite) and that the maximum of the function at <math> x_0 </math> must be a "true" maximum (the number <math> \eta > 0 </math> must exist). There is no need to demand that the integral is finite for <math> n =1 </math> but it is enough to demand that the integral is finite for some <math> n = N </math>.
| |
| | |
| {{cob}}
| |
| | |
| ==Other formulations==
| |
| Laplace's approximation is sometimes written as
| |
| | |
| :<math>\int_a^b\! h(x) e^{M g(x)}\, dx\approx \sqrt{\frac{2\pi}{M|g''(x_0)|}} h(x_0) e^{M g(x_0)} \text { as } M\to\infty \,</math>
| |
| | |
| where <math>h</math> is positive.
| |
| | |
| Importantly, the accuracy of the approximation depends on the variable of integration, that is, on what stays in <math>g(x)</math> and what goes into <math>h(x)</math>.<ref>{{cite book |last=Butler |first=Ronald W |date=2007 |title=Saddlepoint approximations and applications |publisher=Cambridge University Press |isbn=978-0-521-87250-8}}</ref>
| |
| | |
| In the multivariate case where <math>\mathbf{x}</math> is a <math>d</math>-dimensional vector and <math>f(\mathbf{x})</math> is a scalar function of <math>\mathbf{x}</math>, Laplace's approximation is usually written as:
| |
| | |
| :<math>\int e^{M f(\mathbf{x})}\, d\mathbf{x} \approx \left(\frac{2\pi}{M}\right)^{d/2} |H(f)(\mathbf{x}_0)|^{-1/2} e^{M f(\mathbf{x}_0)} \text { as } M\to\infty \,</math> | |
| | |
| where <math>H(f)(\mathbf{x}_0)</math> is the [[Hessian matrix]] of <math>f</math> evaluated at <math>\mathbf{x}_0</math>.
| |
| | |
| ==Laplace's method extension: Steepest descent==
| |
| {{main|method of steepest descent}}
| |
| | |
| In extensions of Laplace's method, [[complex analysis]], and in particular
| |
| [[Cauchy's integral formula]],
| |
| is used to find a contour ''of steepest descent'' for an (asymptotically with large ''M'') equivalent integral, expressed as a [[line integral]]. In particular,
| |
| if no point ''x''<sub>0</sub> where the derivative of ''ƒ'' vanishes exists on the real
| |
| line, it may be necessary to deform the integration contour to an optimal one, where the
| |
| above analysis will be possible. Again the main idea is to reduce, at least asymptotically, the calculation of the given integral to that of a simpler integral that can be explicitly evaluated. See the book of Erdelyi (1956) for a simple discussion (where the method is termed ''steepest descents'').
| |
| | |
| The appropriate formulation for the complex ''z''-plane is
| |
| :<math>\int_a^b\! e^{M f(z)}\, dz\approx \sqrt{\frac{2\pi}{-Mf''(z_0)}}e^{M f(z_0)} \text{ as } M\to\infty. \,</math>
| |
| for a path passing through the saddle point at ''z''<sub>0</sub>.
| |
| Note the explicit appearance of a minus sign to indicate the direction of the second derivative: one must ''not'' take the modulus. Also note that if the integrand is [[meromorphic]], one may have to add residues corresponding to poles traversed while deforming the contour (see for example section 3 of Okounkov's paper ''Symmetric functions and random partitions'').
| |
| | |
| ==Further generalizations==
| |
| An extension of the steepest descent method is the so-called ''nonlinear stationary phase/steepest descent method''. Here, instead of integrals, one needs to evaluate asymptotically solutions of [[Riemann–Hilbert factorization]] problems.
| |
| | |
| Given a contour ''C'' in the [[complex sphere]], a function ''ƒ'' defined on that contour and a special point, say infinity, one seeks a function ''M'' holomorphic away from the contour ''C'', with prescribed jump across ''C'', and with a given normalization at infinity. If ''ƒ'' and hence ''M'' are matrices rather than scalars this is a problem that in general does not admit an explicit solution.
| |
| | |
| An asymptotic evaluation is then possible along the lines of the linear stationary phase/steepest descent method. The idea is to reduce asymptotically the solution of the given Riemann–Hilbert problem to that of a simpler, explicitly solvable, Riemann–Hilbert problem. Cauchy's theorem is used to justify deformations of the jump contour.
| |
| | |
| The nonlinear stationary phase was introduced by Deift and Zhou in 1993, based on earlier work of Its. A (properly speaking) nonlinear steepest descent method was introduced by Kamvissis, K. McLaughlin and P. Miller in 2003, based on previous work of Lax, Levermore, Deift, Venakides and Zhou.
| |
| | |
| The nonlinear stationary phase/steepest descent method has applications to the theory of soliton equations
| |
| and [[integrable model]]s, [[random matrices]] and [[combinatorics]].
| |
| | |
| ==Complex integrals==
| |
| For complex integrals in the form:
| |
| | |
| :<math> \frac{1}{2\pi i}\int_{c-i\infty}^{c+i\infty} g(s)e^{st} \,ds </math>
| |
| | |
| with ''t'' >> 1, we make the substitution ''t'' = ''iu'' and the change of variable ''s'' = ''c'' + ''ix'' to get the Laplace bilateral transform:
| |
| | |
| :<math> \frac{1}{2 \pi}\int_{-\infty}^\infty g(c+ix)e^{-ux}e^{icu} \, dx. </math>
| |
| | |
| We then split ''g''(''c''+''ix'') in its real and complex part, after which we recover ''u'' = ''t'' / ''i''. This is useful for [[inverse Laplace transform]]s, the [[Perron formula]] and complex integration.
| |
| | |
| ==Example 1: Stirling's approximation==
| |
| Laplace's method can be used to derive [[Stirling's approximation]]
| |
| :<math>N!\approx \sqrt{2\pi N} N^N e^{-N}\,</math>
| |
| for a large [[integer]] ''N''.
| |
| | |
| From the definition of the [[Gamma function]], we have
| |
| :<math>N! = \Gamma(N+1)=\int_0^\infty e^{-x} x^N \, dx. </math>
| |
| | |
| Now we change variables, letting
| |
| ::<math>x = N z \,</math>
| |
| so that
| |
| ::<math>dx = N \, dz. </math>
| |
| Plug these values back in to obtain
| |
| | |
| : <math>
| |
| \begin{align}
| |
| N! & = \int_0^\infty e^{-N z} \left(N z \right)^N N \, dz \\
| |
| & = N^{N+1}\int_0^\infty e^{-N z} z^N \, dz \\
| |
| & = N^{N+1}\int_0^\infty e^{-N z} e^{N\ln z} \, dz \\
| |
| & = N^{N+1}\int_0^\infty e^{N(\ln z-z)} \, dz.
| |
| \end{align}
| |
| </math>
| |
| | |
| This integral has the form necessary for Laplace's method with
| |
| :<math>f \left( z \right) = \ln{z}-z </math>
| |
| which is twice-differentiable:
| |
| :<math>f'(z) = \frac{1}{z}-1,\,</math>
| |
| :<math>f''(z) = -\frac{1}{z^2}.\,</math>
| |
| The maximum of ''ƒ''(''z'') lies at ''z''<sub>0</sub> = 1, and the second derivative of ''ƒ''(''z'') has the value −1 at this point. Therefore, we obtain
| |
| :<math>N! \approx N^{N+1}\sqrt{\frac{2\pi}{N}} e^{-N}=\sqrt{2\pi N} N^N e^{-N}.\,</math>
| |
| | |
| ==Example 2: parameter estimation and probabilistic inference==
| |
| {{harvnb|Azevedo-Filho|Shachter|1994}} reviews Laplace's method results ([[univariate]] and [[multivariate]]) and presents a detailed example showing the method used in [[parameter estimation]] and [[probability|probabilistic]] [[inference]] under a [[Bayesian probability|Bayesian]] perspective. Laplace's method is applied to a [[meta-analysis]] problem from the [[medicine|medical]] domain, involving experimental [[data]], and compared to other techniques.
| |
| | |
| ==See also==
| |
| * [[Stationary phase approximation|Method of stationary phase]]
| |
| | |
| ==References==
| |
| {{reflist}}
| |
| | |
| * {{Citation
| |
| | last=Azevedo-Filho
| |
| | first=A.
| |
| | last2=Shachter
| |
| | first2=R.
| |
| | year=1994
| |
| | chapter=Laplace's Method Approximations for Probabilistic Inference in Belief Networks with Continuous Variables
| |
| | editor-first=R.
| |
| | editor-last=Mantaras
| |
| | editor2-first=D.
| |
| | editor2-last=Poole
| |
| | title= Uncertainty in Artificial Intelligence
| |
| | publisher=Morgan Kauffman
| |
| | place=San Francisco, CA
| |
| | id = {{citeseerx|10.1.1.91.2064}}
| |
| }}.
| |
| * {{Citation
| |
| | last=Deift
| |
| | first=P.
| |
| | last2=Zhou
| |
| | first2=X.
| |
| | year=1993
| |
| | title=A steepest descent method for oscillatory Riemann–Hilbert problems. Asymptotics for the MKdV equation
| |
| | periodical=Ann. of Math.
| |
| | volume=137
| |
| | issue=2
| |
| | pages=295–368
| |
| | doi=10.2307/2946540
| |
| }}.
| |
| * {{Citation
| |
| | last=Erdelyi
| |
| | first=A.
| |
| | year=1956
| |
| | title=Asymptotic Expansions
| |
| | publisher=Dover
| |
| }}.
| |
| * {{Citation
| |
| | last=Fog
| |
| | first=A.
| |
| | year=2008
| |
| | title=Calculation Methods for Wallenius' Noncentral Hypergeometric Distribution
| |
| | periodical=Communications in Statistics, Simulation and Computation
| |
| | volume=37
| |
| | issue=2
| |
| | pages=258–273
| |
| | doi=10.1080/03610910701790269
| |
| }}.
| |
| * {{Citation
| |
| | last=Kamvissis
| |
| | first=S.
| |
| | last2=McLaughlin
| |
| | first2=K. T.-R.
| |
| | last3=Miller
| |
| | first3=P.
| |
| | year=2003
| |
| | title=Semiclassical Soliton Ensembles for the Focusing Nonlinear Schrödinger Equation
| |
| | periodical=Annals of Mathematics Studies
| |
| | volume=154
| |
| | publisher=Princeton University Press
| |
| }}.
| |
| * Laplace, P. S. (1774). Memoir on the probability of causes of events. Mémoires de Mathématique et de Physique, Tome Sixième. (English translation by S. M. Stigler 1986. Statist. Sci., 1(19):364–378).
| |
| | |
| {{PlanetMath attribution|id=4284|title=saddle point approximation}}
| |
| | |
| [[Category:Asymptotic analysis]]
| |
| [[Category:Perturbation theory]]
| |