en>Asmeurer: grammar

2013-10-06T20:27:52Z

grammar

← Older revision		Revision as of 22:27, 6 October 2013
Line 1:		Line 1:
	~~They contact me Emilia. For many years he~~'s ~~been working~~ as a ~~meter reader~~ and ~~it's~~ some ~~thing he really enjoy~~. ~~Doing ceramics~~ is ~~what my family members~~ and I ~~enjoy~~. ~~Years~~ in the ~~past we moved~~ to ~~Puerto Rico~~ and ~~my family loves~~ it.<br><br>~~Check out my web blog~~; [http://www.~~adosphere~~.com/~~poyocum at home std test~~]		In [[probability theory]] and [[statistics]], the '''[[Johan Jensen (mathematician)\|Jensen]]–[[Claude Shannon\|Shannon]] divergence''' is a popular method of measuring the similarity between two [[probability distribution]]s. It is also known as '''information radius''' ('''IRad''')<ref>{{cite book \|author=Hinrich Schütze; Christopher D. Manning\|title=Foundations of Statistical Natural Language Processing \|publisher=MIT Press \|location=Cambridge, Mass \|year=1999 \|isbn=0-262-13360-1 \|url=http://nlp.stanford.edu/fsnlp/ \|doi= \|page=304}}</ref> or '''total divergence to the average'''.<ref>{{cite journal\|title=Similarity-Based Methods For Word Sense Disambiguation\|journal=Proceedings of the Thirty-Fifth Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics\|year=1997\|first=Ido\|last=Dagan\|coauthors=Lillian Lee, Fernando Pereira\|volume=\|issue=\|pages=pp. 56–63\|id= \|url=http://citeseer.ist.psu.edu/dagan97similaritybased.html\|accessdate=2008-03-09\|doi=10.3115/979617.979625 }}</ref> It is based on the [[Kullback–Leibler divergence]], with some notable (and useful) differences, including that it is symmetric and it is always a finite value. The square root of the Jensen–Shannon divergence is a [[Metric (mathematics)\|metric]] often referred to as Jensen-Shannon distance.<ref>{{cite journal\|title=A new metric for probability distributions\|journal=IEEE Trans. Inf. Theory\|year=2003\|first=D. M.\|last=Endres\|coauthors=J. E. Schindelin\|volume=49\|issue=7\|pages=pp. 1858–1860\|id= \|url=\|format=\|accessdate=\|doi=10.1109/TIT.2003.813506}}</ref><ref>{{cite journal\|title=A new class of metric divergences on probability spaces and its statistical applications\|journal=Ann. Inst. Statist. Math.\|year=2003\|first=F.\|last=Ôsterreicher\|coauthors=I. Vajda\|volume=55\|issue=3\|pages=pp. 639–653\|id= \|url=\|format=\|accessdate=\|doi=10.1007/BF02517812}}</ref>

			==Definition==
			Consider the set <math>M_+^1(A)</math> of probability distributions where A is a set provided with some [[Sigma-algebra\|σ-algebra]] of measurable subsets. In particular we can take A to be a finite or countable set with all subsets being measurable.

			The Jensen–Shannon divergence (JSD) <math>M_+^1(A) \times M_+^1(A) \rightarrow [0,\infty{})</math> is a symmetrized and smoothed version of the [[Kullback–Leibler divergence]] <math>D(P \parallel Q)</math>. It is defined by

			:<math>{\rm JSD}(P \parallel Q)= \frac{1}{2}D(P \parallel M)+\frac{1}{2}D(Q \parallel M)</math>

			where <math>M=\frac{1}{2}(P+Q)</math>

			If A is countable, a more general definition, allowing for the comparison of more than two distributions, is:

			:<math>{\rm JSD}(P_1, P_2, \ldots, P_n) = H\left(\sum_{i=1}^n \pi_i P_i\right) - \sum_{i=1}^n \pi_i H(P_i)</math>

			where <math>\pi_1, \pi_2, \ldots, \pi_n</math> are the weights for the probability distributions <math>P_1, P_2, \ldots, P_n</math> and <math>H(P)</math> is the [[Shannon entropy]] for distribution <math>P</math>. For the two-distribution case described above,
			:<math>P_1=P, P_2=Q, \pi_1 = \pi_2 = \frac{1}{2}.\ </math>

			==Bounds==
			The Jensen–Shannon divergence is bounded by 1, given that one uses the base 2 logarithm.<ref name=Lin/>

			:<math>0 \leq {\rm JSD}( P \parallel Q ) \leq 1</math>

			For log base e, or ln, which is commonly used in statistical thermodynamics, the upper bound is ln(2):

			:<math>0 \leq {\rm JSD}( P \parallel Q ) \leq \ln(2)</math>

			== Relation to mutual information ==
			Jensen–Shannon divergence is the [[mutual information]] between a random variable <math>X</math> from a [[mixture distribution]] <math>M=\frac{P+Q}{2}</math> and a binary indicator variable <math>Z</math> where <math>Z = 1</math> if <math>X</math> is from <math>P</math> and <math>Z = 0</math> if <math>X</math> is from <math>Q</math>.
			:<math>\begin{align}
			I(X; Z) &= H(X) - H(X\|Z)\\
			&= -\sum M \log M + \frac{1}{2} \left[ \sum P \log P + \sum Q \log Q \right] \\
			&= -\sum \frac{P}{2} \log M - \sum \frac{Q}{2} \log M + \frac{1}{2} \left[ \sum P \log P + \sum Q \log Q \right] \\
			&= \frac{1}{2} \sum P \left( \log P - \log M\right ) + \frac{1}{2} \sum Q \left( \log Q - \log M \right) \\
			&= {\rm JSD}(P \parallel Q)
			\end{align}
			</math>
			It follows from the above result that Jensen–Shannon divergence is bounded by 0 and 1 because mutual information is non-negative and bounded by <math>H(Z) = 1</math>.

			One can apply the same principle to the joint and product of marginal distribution (in analogy to Kullback–Leibler divergence and mutual information) and to measure how reliably one can decide if a given response comes from the joint distribution or the product distribution—given that these are the only possibilities.<ref>{{cite journal\|title=Synergy, Redundancy, and Independence in Population Codes\|journal=Journal of Neuroscience\|year=2003\|first=Elad\|last=Schneidman\|pages=11539–11553\|volume=23\|issue=37\|pmid=14684857\|last2=Bialek\|first2=W\|last3=Berry\|first3=M.J. 2nd}}</ref>

			==Quantum Jensen–Shannon divergence==
			The generalization of probability distributions on [[density matrices]] allows to define quantum Jensen–Shannon divergence (QJSD).<ref>{{cite journal\|doi=10.1103/PhysRevA.72.052310\|title=Jensen-Shannon divergence as a measure of distinguishability between mixed quantum states\|year=2005\|last1=Majtey\|first1=A.\|last2=Lamberti\|first2=P.\|last3=Prato\|first3=D.\|journal=Physical Review A\|volume=72\|issue=5}}</ref><ref name="briet">{{Cite journal\|doi=10.1103/PhysRevA.79.052311\|title=Properties of classical and quantum Jensen-Shannon divergence\|year=2009\|last1=Briët\|first1=Jop\|last2=Harremoës\|first2=Peter\|journal=Physical Review A\|volume=79\|issue=5}}</ref> It is defined for a set of [[density matrices]] <math>(\rho_1,\ldots,\rho_n)</math> and probability distribution <math>\pi=(\pi_1,\ldots,\pi_n)</math> as

			:<math>{\rm QJSD}(\rho_1,\ldots,\rho_n)= S\left(\sum_{i=1}^n \pi_i \rho_i\right)-\sum_{i=1}^n \pi_i S(\rho_i)</math>

			where <math>S(\rho)</math> is the [[von Neumann entropy]]. This quantity was introduced in quantum information theory, where it is called the Holevo information: it gives the upper bound for amount of classical information encoded by the quantum states <math>(\rho_1,\ldots,\rho_n)</math> under the prior distribution <math>\pi</math> (see [[Holevo's theorem]])<ref>{{citation\|first=A. S.\|last=Holevo\|authorlink=Alexander Holevo\|year=1973\|title=Bounds for the quantity of information transmitted by a quantum communication channel\|journal=Problemy Peredachi Informatsii\|volume=9\|pages=3–11\|year=1973\|language=Russian}}. English translation: ''Probl. Inf. Transm''., 9, 177–183 (1975)) {{mr\|456936}}</ref> Quantum Jensen–Shannon divergence for <math>\pi=\left(\frac{1}{2},\frac{1}{2}\right)</math> and two density matrices is a symmetric function, everywhere defined, bounded and equal to zero only if two [[density matrices]] are the same. It is a square of a metric for [[pure states]]<ref>{{cite journal\|doi=10.1103/PhysRevLett.72.3439\|title=Statistical distance and the geometry of quantum states\|year=1994\|last1=Braunstein\|first1=Samuel\|last2=Caves\|first2=Carlton\|journal=Physical Review Letters\|volume=72\|issue=22\|pages=3439–3443\|pmid=10056200}}</ref> but it is unknown whether the metric property holds in general.<ref name="briet" /> The [[Bures metric]] is closely related to the quantum JS divergence; it is the quantum analog of the [[Fisher information metric]].

			==Applications==
			The Jensen–Shannon divergence has been applied in [[bioinformatics]] and [[genome comparison]],<ref name=Sims/><ref name=It/> and in protein surface comparison.<ref name=Ofran/>

			== Notes ==
			{{reflist\|refs=

			<ref name=Lin>{{cite journal
			\|first=J. \|last=Lin
			\|url=https://www.cise.ufl.edu/~anand/sp06/jensen-shannon.pdf
			\|title=Divergence measures based on the shannon entropy
			\|journal=IEEE Transactions on Information Theory
			\|volume=37 \|issue=1 \|pages=145–151
			\|year=1991 \|doi=10.1109/18.61115}}</ref>

			<ref name=Sims>{{Cite journal\|pmid=19188606\|year=2009\|last1=Sims\|first1=GE\|last2=Jun\|first2=SR\|last3=Wu\|first3=GA\|last4=Kim\|first4=SH\|title=Alignment-free genome comparison with feature frequency profiles (FFP) and optimal resolutions\|volume=106\|issue=8\|pages=2677–82\|doi=10.1073/pnas.0813249106\|pmc=2634796\|journal=Proceedings of the National Academy of Sciences of the United States of America }}</ref>

			<ref name=It>{{Cite journal\|pmid=20841429\|year=2010\|last1=Itzkovitz\|first1=S\|last2=Hodis\|first2=E\|last3=Segal\|first3=E\|title=Overlapping codes within protein-coding sequences\|volume=20\|issue=11\|pages=1582–9\|doi=10.1101/gr.105072.110\|pmc=2963821\|journal=Genome research}}</ref>

			<ref name=Ofran>{{Cite journal\|pmid=12488102\|year=2003\|last1=Ofran\|first1=Y\|last2=Rost\|first2=B\|title=Analysing six types of protein-protein interfaces\|volume=325\|issue=2\|pages=377–87\|journal=Journal of molecular biology}}</ref>
			}}

			== Further reading ==
			*{{Cite journal\|url=http://www.math.ku.dk/~topsoe/ISIT2004JSD.pdf\|doi=10.1109/ISIT.2004.1365067\|chapter=Jensen-Shannon divergence and Hilbert space embedding\|title=International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings\|year=2004\|last1=Fuglede\|first1=B.\|last2=Topsoe\|first2=F.\|isbn=0-7803-8280-3\|pages=30}}
			* {{cite arxiv\|eprint=1009.4004\|author1=Frank Nielsen\|title=A family of statistical symmetric divergences based on Jensen's inequality\|class=cs.CV\|year=2010}}

			==External links==
			* [https://github.com/evansenter/diverge Ruby gem for calculating JS divergence]
			* [http://pynchon.googlecode.com/hg/pynchon/bio/alg/entropy.py Python code for calculating JS divergence]

			{{DEFAULTSORT:Jensen-Shannon Divergence}}
			[[Category:Statistical distance measures]]

en>Cydebot: Robot - Moving category Constraint satisfaction to :Category:Constraint programming per CFD at Wikipedia:Categories for discussion/Log/2011 June 15.

2011-06-30T16:23:30Z

Robot - Moving category Constraint satisfaction to Category:Constraint programming per CFD at Wikipedia:Categories for discussion/Log/2011 June 15.

New page

They contact me Emilia. For many years he's been working as a meter reader and it's some thing he really enjoy. Doing ceramics is what my family members and I enjoy. Years in the past we moved to Puerto Rico and my family loves it.<br><br>Check out my web blog; [http://www.adosphere.com/poyocum at home std test]

Constraint learning - Revision history

en>Asmeurer: grammar

en>Cydebot: Robot - Moving category Constraint satisfaction to :Category:Constraint programming per CFD at Wikipedia:Categories for discussion/Log/2011 June 15.