latentspace.

KL-Divergence


  • KL-divergence
    • In mathematical statistics, the Kullback-Leibler (KL) divergence (also called relative entropy and I-divergence), denoted \(\text{D}_{KL} (P \, \| \, Q) \), is a type of statistical distance: a measure of how much a model probability distribution \(Q\) is different from a true probability distribution \(P\).
    • While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance.
    • Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical.
    • For discrete probability distributions \(P\) and \(Q\) defined on the same sample space, \(\mathcal{X}\), the KL-divergence from \(Q\) to \(P\) is defined to be: \[ \,\\ \text{D}_{KL} (P \, \| \, Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right) \,\\ \]

    • For continuous probability distributions, the sum is replaced by an integral, where \(p\) and \(q\) denote the probability densities of \(P\) and \(Q\) respectively. \[ \,\\ \text{D}_{KL} (P \, \| \, Q) = \displaystyle\int\limits_{\qquad-\infty}^{\qquad\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right) dx \,\\ \]