Topos

In information theory and statistics, Kullback's inequality is a lower bound on the Kullback–Leibler divergence expressed in terms of the large deviations rate function. If P and Q are probability distributions on the real line, such that P is absolutely continuous with respect to Q, i.e. P<<Q, and whose first moments exist, then

$D_{KL}(P\|Q)\geq \Psi _{Q}^{*}(\mu '_{1}(P)),$

where $\Psi _{Q}^{*}$ is the rate function, i.e. the convex conjugate of the cumulant-generating function, of $Q$, and $\mu '_{1}(P)$ is the first moment of $P.$

The Cramér–Rao bound is a corollary of this result.

Proof

Let P and Q be probability distributions (measures) on the real line, whose first moments exist, and such that P<<Q. Consider the natural exponential family of Q given by

$Q_{\theta }(A)={\frac {\int _{A}e^{\theta x}Q(dx)}{\int _{-\infty }^{\infty }e^{\theta x}Q(dx)}}={\frac {1}{M_{Q}(\theta )}}\int _{A}e^{\theta x}Q(dx)$

for every measurable set A, where $M_{Q}$ is the moment-generating function of Q. (Note that Q₀=Q.) Then

$D_{KL}(P\|Q)=D_{KL}(P\|Q_{\theta })+\int _{\mathrm {supp} P}\left(\log {\frac {\mathrm {d} Q_{\theta }}{\mathrm {d} Q}}\right)\mathrm {d} P.$

By Gibbs' inequality we have $D_{KL}(P\|Q_{\theta })\geq 0$ so that

$D_{KL}(P\|Q)\geq \int _{\mathrm {supp} P}\left(\log {\frac {\mathrm {d} Q_{\theta }}{\mathrm {d} Q}}\right)\mathrm {d} P=\int _{\mathrm {supp} P}\left(\log {\frac {e^{\theta x}}{M_{Q}(\theta )}}\right)P(dx)$

Simplifying the right side, we have, for every real θ where $M_{Q}(\theta )<\infty :$

$D_{KL}(P\|Q)\geq \mu '_{1}(P)\theta -\Psi _{Q}(\theta ),$

where $\mu '_{1}(P)$ is the first moment, or mean, of P, and $\Psi _{Q}=\log M_{Q}$ is called the cumulant-generating function. Taking the supremum completes the process of convex conjugation and yields the rate function:

$D_{KL}(P\|Q)\geq \sup _{\theta }\left\{\mu '_{1}(P)\theta -\Psi _{Q}(\theta )\right\}=\Psi _{Q}^{*}(\mu '_{1}(P)).$

Corollary: the Cramér–Rao bound

Mining Engineer (Excluding Oil ) Truman from Alma, loves to spend time knotting, largest property developers in singapore developers in singapore and stamp collecting. Recently had a family visit to Urnes Stave Church.

Start with Kullback's inequality

Let X_θ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then

$\lim _{h\rightarrow 0}{\frac {D_{KL}(X_{\theta +h}\|X_{\theta })}{h^{2}}}\geq \lim _{h\rightarrow 0}{\frac {\Psi _{\theta }^{*}(\mu _{\theta +h})}{h^{2}}},$

where $\Psi _{\theta }^{*}$ is the convex conjugate of the cumulant-generating function of $X_{\theta }$ and $\mu _{\theta +h}$ is the first moment of $X_{\theta +h}.$

Left side

The left side of this inequality can be simplified as follows:

$\lim _{h\rightarrow 0}{\frac {D_{KL}(X_{\theta +h}\|X_{\theta })}{h^{2}}}=\lim _{h\rightarrow 0}{\frac {1}{h^{2}}}\int _{-\infty }^{\infty }\left(\log {\frac {\mathrm {d} X_{\theta +h}}{\mathrm {d} X_{\theta }}}\right)\mathrm {d} X_{\theta +h}$

$=\lim _{h\rightarrow 0}{\frac {1}{h^{2}}}\int _{-\infty }^{\infty }\left[\left(1-{\frac {\mathrm {d} X_{\theta }}{\mathrm {d} X_{\theta +h}}}\right)+{\frac {1}{2}}\left(1-{\frac {\mathrm {d} X_{\theta }}{\mathrm {d} X_{\theta +h}}}\right)^{2}+o\left(\left(1-{\frac {\mathrm {d} X_{\theta }}{\mathrm {d} X_{\theta +h}}}\right)^{2}\right)\right]\mathrm {d} X_{\theta +h},$

where we have expanded the logarithm $\log x$ in a Taylor series in $1-1/x$,

$=\lim _{h\rightarrow 0}{\frac {1}{h^{2}}}\int _{-\infty }^{\infty }\left[{\frac {1}{2}}\left(1-{\frac {\mathrm {d} X_{\theta }}{\mathrm {d} X_{\theta +h}}}\right)^{2}\right]\mathrm {d} X_{\theta +h}$

$=\lim _{h\rightarrow 0}{\frac {1}{h^{2}}}\int _{-\infty }^{\infty }\left[{\frac {1}{2}}\left({\frac {\mathrm {d} X_{\theta +h}-\mathrm {d} X_{\theta }}{\mathrm {d} X_{\theta +h}}}\right)^{2}\right]\mathrm {d} X_{\theta +h}={\frac {1}{2}}{\mathcal {I}}_{X}(\theta ),$

which is half the Fisher information of the parameter θ.

Right side

The right side of the inequality can be developed as follows:

$\lim _{h\rightarrow 0}{\frac {\Psi _{\theta }^{*}(\mu _{\theta +h})}{h^{2}}}=\lim _{h\rightarrow 0}{\frac {1}{h^{2}}}{\sup _{t}\{\mu _{\theta +h}t-\Psi _{\theta }(t)\}}.$

This supremum is attained at a value of t=τ where the first derivative of the cumulant-generating function is $\Psi '_{\theta }(\tau )=\mu _{\theta +h},$ but we have $\Psi '_{\theta }(0)=\mu _{\theta },$ so that

$\Psi ''_{\theta }(0)={\frac {d\mu _{\theta }}{d\theta }}\lim _{h\rightarrow 0}{\frac {h}{\tau }}.$

Moreover,

$\lim _{h\rightarrow 0}{\frac {\Psi _{\theta }^{*}(\mu _{\theta +h})}{h^{2}}}={\frac {1}{2\Psi ''_{\theta }(0)}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2}={\frac {1}{2\mathrm {Var} (X_{\theta })}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2}.$

Putting both sides back together

We have:

${\frac {1}{2}}{\mathcal {I}}_{X}(\theta )\geq {\frac {1}{2\mathrm {Var} (X_{\theta })}}\left({\frac {d\mu _{\theta }}{d\theta }}\right)^{2},$

which can be rearranged as:

$\mathrm {Var} (X_{\theta })\geq {\frac {(d\mu _{\theta }/d\theta )^{2}}{{\mathcal {I}}_{X}(\theta )}}.$

See also

• Kullback–Leibler divergence
• Cramér–Rao bound
• Fisher information
• Large deviations theory
• Convex conjugate
• Rate function
• Moment-generating function