Summarising the DDPM research paper

15 Jan, 2025

I've been fascinated by the recent explosion of AI art generation models, and after diving deep into the research, I wanted to share my understanding of one of the foundational papers that made it all possible - Denoising Diffusion Probabilistic Models (DDPM). Fair warning: we'll get into some math, but I'll try to keep things as intuitive as possible.

So what got me hooked on this paper? Well, it's pretty mind-blowing how these models can start with pure noise and gradually transform it into incredibly detailed images. The core idea behind DDPM is actually inspired by thermodynamics - specifically, how particles diffuse from areas of high concentration to low concentration. The researchers had this brilliant insight to reverse this natural process for generating images.

Let me break down how it works. Imagine you have an image, and you slowly add random noise to it until it becomes completely random - that's the forward diffusion process. The clever part is training a neural network to learn how to reverse this process, starting from random noise and gradually removing it to create a coherent image.

The math behind this is pretty elegant. The forward process is defined as a Markov chain that gradually adds Gaussian noise to the image. For each step $t$, we have:

where ${\beta }_t$ is a carefully chosen schedule of noise levels that increases over time. What's cool is that we can actually skip steps in this process using a neat mathematical trick. We can directly sample $x_t$ given $x_0$ using:

where ${\alpha }_t=1-{\beta }_t$ and ${\overline{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$.

Now, the real magic happens in the reverse process. We train a neural network to predict the noise that was added at each step, which lets us gradually denoise the image. The reverse process is also a Markov chain:

When I first implemented this, I was surprised by how well it worked even with a relatively simple U-Net architecture. The training objective is basically just minimizing the difference between the predicted and actual noise at each step:

What I find particularly fascinating is how this relatively simple idea leads to such high-quality results. Unlike GANs, which can be notoriously unstable to train (trust me, I've been there), DDPMs are much more stable and consistently produce good results. The trade-off is that they're slower at generation time because you need to iterate through multiple denoising steps.

One thing that really clicked for me was understanding that the model isn't trying to generate an image directly - instead, it's learning the much simpler task of predicting noise. This makes the learning problem more tractable and leads to better results.

I've been experimenting with different noise schedules, and I've found that using a cosine schedule as proposed in improved-DDPM works really well. It helps maintain image quality throughout the diffusion process and seems to lead to better samples.

The impact of this paper has been pretty incredible. It laid the groundwork for models like Stable Diffusion and DALL-E 2, which have revolutionized how we think about AI art generation. While newer papers have built upon and improved these ideas, the core principles from DDPM remain fundamental to how these systems work.

If you're interested in implementing this yourself, I'd recommend starting with a simple U-Net on MNIST or CIFAR-10. It's a great way to get a feel for how the process works, and there's something magical about watching random noise gradually transform into recognizable images.

Looking back, it's amazing how this paper managed to make such a complex idea both mathematically elegant and practically implementable. It's a beautiful example of how insights from physics can lead to breakthroughs in machine learning.