In mathematical statistics, the Kullback-Leibler (KL) divergence (also called relative entropy and I-divergence), denoted
\(\text{D}_{KL} (P \, \| \, Q) \), is a type of statistical distance:
a measure of how much a model probability distribution \(Q\) is different from a true probability distribution \(P\).
While it is a measure of how different two distributions are, and in some sense is thus a
"distance", it is not actually a metric, which is the most familiar and formal type of distance.
Relative entropy is always a non-negative real number,
with value 0 if and only if the two distributions in question are identical.
For discrete probability distributions \(P\) and \(Q\) defined on the same sample space, \(\mathcal{X}\),
the KL-divergence from \(Q\) to \(P\) is defined to be:
\[
\,\\
\text{D}_{KL} (P \, \| \, Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right)
\,\\
\]
For continuous probability distributions, the sum is replaced by an integral,
where \(p\) and \(q\) denote the probability densities of \(P\) and \(Q\) respectively.
\[
\,\\
\text{D}_{KL} (P \, \| \, Q) = \displaystyle\int\limits_{\qquad-\infty}^{\qquad\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right) dx
\,\\
\]