KL-Divergence 12/17/2024 KL-divergence In mathematical statistics, the Kullback-Leibler (KL) divergence (also called relative entropy and I-divergence), denoted \(\text{D}_{KL} (P \, \| \, Q) \), is a type of statistical distance: a measure of how much a model probability distribution \(Q\) is different from a true probability distribution \(P\). While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical. For discrete probability distributions \(P\) and \(Q\) defined on the same sample space, \(\mathcal{X}\), the KL-divergence from \(Q\) to \(P\) is defined to be: \[ \,\\ \text{D}_{KL} (P \, \| \, Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right) \,\\ \] For continuous probability distributions, the sum is replaced by an integral, where \(p\) and \(q\) denote the probability densities of \(P\) and \(Q\) respectively. \[ \,\\ \text{D}_{KL} (P \, \| \, Q) = \displaystyle\int\limits_{\qquad-\infty}^{\qquad\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right) dx \,\\ \] References https://angeloyeo.github.io/2020/10/27/KL_divergence.html https://bigdatamaster.tistory.com/153