Intuitive understanding of KL Divergence

14 Jan, 2025

Hmm sooo KL Divergence is what we have to walk through, and I had this interesting realization while playing with some dice simulations. You know how we often think of KL divergence as this abstract information-theoretic concept? Well, it turns out there's this really neat connection to something more intuitive: probability ratios of seeing the same sequence of observations under different models.

To start, the mathematical definition of KL divergence for discrete distributions is:

D_{K L} (P | Q) = \sum_{x \in X} P (x) \log \frac{P (x)}{Q (x)}

where $X$ is the set of all possible events, $P (x)$ is the probability of event $x$ under the true distribution, and $Q (x)$ is the probability of the same event under the approximating distribution.

Huhhhhh !! what is this ? sounds pretty vague right???

Let me break this down with a concrete example. Imagine you have two dice - one fair and one loaded. The fair die represents your "true" distribution P, and the loaded die represents your "model" distribution Q. Now, let's say you roll these dice many times and want to compare how likely you are to see the exact same sequence of numbers under each die.

For a single roll, the probability ratio would be:

$\frac{P (x)}{Q (x)}$

But what about for multiple independent rolls? Well, for n rolls, we'd multiply these ratios:

\prod_{i = 1}^{n} \frac{P (x_{i})}{Q (x_{i})}

Now here's where it gets interesting. Taking the log of this ratio (because logs make everything better, right?), we get:

\log (\prod_{i = 1}^{n} \frac{P (x_{i})}{Q (x_{i})}) = \sum_{i = 1}^{n} \log \frac{P (x_{i})}{Q (x_{i})}

As n gets large, thanks to the law of large numbers, this sum divided by n approaches the expected value:

\frac{1}{n} \sum_{i = 1}^{n} \log \frac{P (x_{i})}{Q (x_{i})} \to E_{P} [\log \frac{P (X)}{Q (X)}]

And guess what? This expectation is exactly the KL divergence from Q to P:

D_{K L} (P | | Q) = E_{P} [\log \frac{P (X)}{Q (X)}] = \sum_{x} P (x) \log \frac{P (x)}{Q (x)}

This means that KL divergence is essentially telling us how easy it is to distinguish between two distributions by looking at sequences of observations. A larger KL divergence means the probability ratio of seeing the same sequence under both distributions becomes exponentially small as we collect more data.

Let's make this concrete with our dice example. Say our loaded die has probabilities: $Q (1) = 0.3, Q (2) = 0.2, Q (3) = Q (4) = Q (5) = Q (6) = 0.125$

While the fair die has $P (x) = \frac{1}{6}$ for all x.

The KL divergence here would be:

D_{K L} (P | | Q) = \sum_{i = 1}^{6} \frac{1}{6} \log \frac{1 / 6}{Q (i)} \approx 0.0823

This seemingly small number has a profound meaning: if you roll both dice n times, the probability of seeing the same sequence under both distributions is approximately $e^{- 0.0823 n}$ . After just 100 rolls, this probability becomes vanishingly small!

I find this connection fascinating because it gives us an intuitive way to think about KL divergence. It's not just some abstract measure - it's directly related to how quickly we can distinguish between two distributions by observing data. The larger the KL divergence, the faster we can tell them apart.

This also explains why KL divergence is so useful in machine learning. When we're training models, we're essentially trying to make our model distribution Q as close as possible to the true data distribution P. The KL divergence gives us a meaningful measure of how well we're doing, grounded in the practical reality of observing data.

Next time you're working with KL divergence, remember this interpretation. It's not just pushing around symbols - it's measuring something fundamental about how distributions differ in terms of the sequences they generate. Pretty cool, right?