From TheBestLinks.com
In probability theory, the Kullback-Leibler divergence is a quantity which measures the difference between two probability distributions.
The term "divergence" is a misnomer;
it is not the same as divergence in calculus.
One might be tempted to call it a "distance",
but this would also be a misnomer as the Kullback-Leibler divergence is not symmetric.
The Kullback-Leibler divergence between two probability distributions p and q is defined as
- <math> KL(p,q) = \sum_x p(x) \log\frac{p(x)}{q(x)} <math>
for distributions of a discrete variable,
and as
- <math> KL(p,q) = \int_{-\infty}^{\infty} p(x) \log\frac{p(x)}{q(x)} \; dx <math>
for distributions of a continuous variable.
It can be seen from the definition that
- <math> KL(p,q) = -\sum_x p(x) \log q(x) + \sum_x p(x) \log p(x)
= H(p,q) - H(p)\, <math>
denoting by H(p,q) the cross-entropy of p and q,
and by H(p) the entropy of p.
As the cross-entropy is always greater than or equal to the entropy,
this shows that the Kullback-Leibler divergence is nonnegative,
and furthermore KL(p,p) is zero.
Related links
Top visited
0 of
0 links
[no links posted yet]
>> place link >>
Discussion
Last posted
0 of
0 messages
[no messages posted yet]
>> post message >>
Watch
You can
add this article to your own "watchlist" and receive e-mail notification about all changes in this page.
