GRPO explained in depth.

07 Jun, 2025

diving into why group relative policy optimization (grpo) is needed

hi there! let’s dive deep into group relative policy optimization (grpo), a reinforcement learning (rl) method that’s been a game-changer, especially for fine-tuning large language models (llms) on tasks like reasoning and math problems. if you’re new to rl or need a refresher, i’d suggest checking out this post: Proximal Policy Optimisation. it’ll get you up to speed so we can focus entirely on grpo here.

this is going to be a long, detailed journey, and i’ll walk you through it like we’re chatting over coffee. we’ll explore why grpo is necessary, building up the concepts brick by brick, deriving every equation step by step, and weaving in the reasoning behind each piece without explicitly labeling it as intuition. my goal is to make the math feel natural and motivated, as if we’re discovering it together. let’s get started!

the challenge of optimizing policies in rl

in rl, we’re training an agent—like an llm generating text—to make decisions that maximize a total reward over time. the agent follows a policy, a strategy that picks an action (like the next token to generate) based on the current state (like the prompt and text so far). this policy is a neural network with parameters $θ$ , written as $π_{θ} (a | s)$ , which gives the probability of choosing action $a$ in state $s$ .

picture a robot navigating a maze. at each time step $t$ , it’s in state $s_{t}$ , picks an action $a_{t}$ using its policy, receives a reward $r_{t}$ , and moves to state $s_{t + 1}$ . the robot’s goal is to rack up as much reward as possible over time. our job is to tweak the policy’s parameters $θ$ to maximize the expected total reward, which we’ll call the objective $J (θ)$ . in policy gradient methods, we compute the gradient of $J (θ)$ with respect to $θ$ and use gradient ascent to adjust $θ$ in the direction that increases the reward. but this process is tricky, especially for something as complex as an llm. let’s break down the challenges by starting with the basics.

vanilla policy gradients: the starting point

let’s begin with the simplest policy gradient method to understand where things go wrong. our objective $J (θ)$ is the expected total reward starting from an initial state, following the policy $π_{θ}$ . imagine the robot starting at the maze entrance—we want to know the average total reward it can expect by following $π_{θ}$ . we can write this as:

J (θ) = 𝔼_{π} [G_{0}]

here, $G_{0}$ is the total discounted reward starting from the initial state $s_{0}$ . the robot collects rewards $r_{0}, r_{1}, r_{2}, \dots$ at each step, but future rewards are worth less than immediate ones—think of getting $1 today versus $1 next year. to account for this, we use a discount factor $γ$ (between 0 and 1), so:

G_{0} = r_{0} + γ r_{1} + γ^{2} r_{2} + \dots = \sum_{t = 0}^{\infty} γ^{t} r_{t}

the expectation $𝔼_{π}$ is taken over all possible trajectories (sequences of states, actions, and rewards) the robot might experience under $π_{θ}$ . a trajectory $τ$ is a sequence $(s_{0}, a_{0}, r_{0}, s_{1}, a_{1}, r_{1}, \dots)$ , and its probability depends on the policy $π_{θ}$ and the environment’s dynamics. the probability of a trajectory is:

p (τ ∣ θ) = p (s_{0}) \prod_{t = 0}^{\infty} π_{θ} (a_{t} ∣ s_{t}) p (s_{t + 1} ∣ s_{t}, a_{t})

where $p (s_{0})$ is the initial state distribution, $π_{θ} (a_{t} ∣ s_{t})$ is the policy’s probability of choosing $a_{t}$ in $s_{t}$ , and $p (s_{t + 1} ∣ s_{t}, a_{t})$ is the environment’s transition probability. so we can rewrite $J (θ)$ as:

J (θ) = \int p (τ ∣ θ) G (τ) d τ

where $G (τ)$ is the total discounted reward for trajectory $τ$ . to improve the policy, we need the gradient $\nabla_{θ} J (θ)$ . since $p (τ ∣ θ)$ depends on $θ$ through $π_{θ}$ , we need to differentiate this integral. let’s compute the gradient:

\nabla_{θ} J (θ) = \nabla_{θ} \int p (τ ∣ θ) G (τ) d τ = \int \nabla_{θ} p (τ ∣ θ) G (τ) d τ

since $G (τ)$ doesn’t depend on $θ$ , we focus on $\nabla_{θ} p (τ ∣ θ)$ . there’s a handy trick here: notice that $\nabla_{θ} p (τ ∣ θ) = p (τ ∣ θ) \nabla_{θ} \log p (τ ∣ θ)$ , because:

\nabla_{θ} \log p (τ ∣ θ) = \frac{\nabla_{θ} p (τ ∣ θ)}{p (τ ∣ θ)}

substituting this in, we get:

\nabla_{θ} J (θ) = \int p (τ ∣ θ) (\nabla_{θ} \log p (τ ∣ θ)) G (τ) d τ = 𝔼_{π} [\nabla_{θ} \log p (τ ∣ θ) G (τ)]

now, let’s compute $\nabla_{θ} \log p (τ ∣ θ)$ . first, take the log of the trajectory probability:

\log p (τ ∣ θ) = \log p (s_{0}) + \sum_{t = 0}^{\infty} (\log π_{θ} (a_{t} ∣ s_{t}) + \log p (s_{t + 1} ∣ s_{t}, a_{t}))

differentiate with respect to $θ$ :

\nabla_{θ} \log p (τ ∣ θ) = \nabla_{θ} (\log p (s_{0}) + \sum_{t = 0}^{\infty} \log π_{θ} (a_{t} ∣ s_{t}) + \sum_{t = 0}^{\infty} \log p (s_{t + 1} ∣ s_{t}, a_{t}))

the terms $\log p (s_{0})$ and $\log p (s_{t + 1} ∣ s_{t}, a_{t})$ don’t depend on $θ$ , so their gradients are zero. the only terms that depend on $θ$ are the policy terms:

\nabla_{θ} \log p (τ ∣ θ) = \sum_{t = 0}^{\infty} \nabla_{θ} \log π_{θ} (a_{t} ∣ s_{t})

plug this back into the gradient:

\nabla_{θ} J (θ) = 𝔼_{π} [(\sum_{t = 0}^{\infty} \nabla_{θ} \log π_{θ} (a_{t} ∣ s_{t})) G (τ)]

in practice, we approximate this by using $G_{t}$ , the total discounted reward from time $t$ onward:

G_{t} = r_{t} + γ r_{t + 1} + γ^{2} r_{t + 2} + \dots = \sum_{k = 0}^{\infty} γ^{k} r_{t + k}

this leads to the policy gradient form:

\nabla_{θ} J (θ) = 𝔼_{π} [\sum_{t = 0}^{\infty} \nabla_{θ} \log π_{θ} (a_{t} ∣ s_{t}) G_{t}]

to use this, we sample trajectories, compute $G_{t}$ for each step, and update $θ$ to make high-reward actions more likely. but there’s a problem: $G_{t}$ can vary a lot between trajectories. one path might get a huge reward by chance, while another gets almost nothing, even if the actions were similar. this high variance makes the gradient noisy, causing the robot to overcorrect—like always turning left because one left turn got lucky. also, a single high-reward trajectory can cause a massive update to $θ$ , drastically changing the policy and potentially ruining performance. we need to reduce this variance and control how much the policy changes.

reinforce with a baseline: tackling variance

let’s improve this by adding a baseline to reduce variance, leading to the reinforce algorithm with a baseline. instead of using $G_{t}$ directly, we subtract a baseline to compute the advantage. the idea is to measure how much better an action is compared to the average, rather than looking at the raw reward. we define the advantage as:

A_{t} = G_{t} - b (s_{t})

here, $b (s_{t})$ is a baseline, often chosen as the expected reward in state $s_{t}$ , which we approximate with a value function $V (s_{t})$ that estimates the average future reward starting from $s_{t}$ under $π_{θ}$ . the policy gradient becomes:

\nabla_{θ} J (θ) = 𝔼_{π} [\sum_{t = 0}^{\infty} \nabla_{θ} \log π_{θ} (a_{t} ∣ s_{t}) A_{t}]

let’s see why this reduces variance. without the baseline, our update term is $\nabla_{θ} \log π_{θ} (a_{t} ∣ s_{t}) G_{t}$ . the variance of this term depends on $G_{t}$ , which can swing widely. for example, if $G_{t}$ ranges from -100 to 100 across trajectories, the updates will be all over the place. now, with the baseline, we use $A_{t} = G_{t} - b (s_{t})$ . if we choose $b (s_{t})$ to be close to the expected value of $G_{t}$ in state $s_{t}$ , i.e., $b (s_{t}) \approx 𝔼_{π} [G_{t} ∣ s_{t}]$ , then:

𝔼_{π} [A_{t} ∣ s_{t}] = 𝔼_{π} [G_{t} ∣ s_{t}] - b (s_{t}) \approx 0

this centers $A_{t}$ around 0, reducing its fluctuations. to quantify this, let’s compare the variances. the variance of $G_{t}$ is:

Var (G_{t} ∣ s_{t}) = 𝔼_{π} [G_{t}^{2} ∣ s_{t}] - (𝔼_{π} [G_{t} ∣ s_{t}])^{2}

now, the variance of $A_{t}$ :

Var (A_{t} ∣ s_{t}) = 𝔼_{π} [(G_{t} - b (s_{t}))^{2} ∣ s_{t}] - (𝔼_{π} [G_{t} - b (s_{t})] ∣ s_{t})^{2}

since $𝔼_{π} [G_{t} - b (s_{t})] ∣ s_{t} \approx 0$ , the second term is near 0, and the first term, $𝔼_{π} [(G_{t} - b (s_{t}))^{2} ∣ s_{t}]$ , is the variance of $G_{t}$ around $b (s_{t})$ , which is smaller than the variance of $G_{t}$ around its mean. this makes the updates more stable, as $A_{t}$ doesn’t swing as much as $G_{t}$ .

we estimate $V (s_{t})$ using a separate neural network, called the critic, which we train to predict the expected reward. but this comes with challenges. the critic is another network, often as big as the policy network, doubling our memory and compute needs. for an llm with billions of parameters, this is a major issue—imagine needing twice the gpu memory! also, if the critic’s estimate $V (s_{t})$ is off, $A_{t}$ becomes misleading, leading to poor updates. training the critic is tough, especially in llm tasks where rewards are sparse, like in math problems where only the final answer gets a reward, and earlier steps get nothing. we need a way to compute advantages without a critic, as it’s too resource-heavy and error-prone.

proximal policy optimization (ppo): controlling policy changes

let’s look at ppo, a method that addresses the issue of large policy updates. ppo uses a trust region to ensure we don’t change the policy too much at once. the idea is to keep the new policy close to the old one, so we don’t risk a performance drop—like the robot suddenly changing its entire maze strategy and getting lost.

ppo compares the new policy $π_{θ}$ to the old policy $π_{θ_{old}}$ (before the update) using a probability ratio:

r_{t} (θ) = \frac{π_{θ} (a_{t} ∣ s_{t})}{π_{θ_{old}} (a_{t} ∣ s_{t})}

if $r_{t} (θ) > 1$ , the new policy makes $a_{t}$ more likely; if $r_{t} (θ) < 1$ , it makes $a_{t}$ less likely. a natural objective would be to maximize:

L (θ) = 𝔼_{t} [r_{t} (θ) A_{t}]

this approximates $J (θ)$ locally, because for small changes in $θ$ , $r_{t} (θ) \approx 1$ , and the gradient of this objective matches the policy gradient. but if $r_{t} (θ)$ gets too large or too small, the update can be too aggressive. let’s derive this issue. if $A_{t} > 0$ , we want to increase $π_{θ} (a_{t} ∣ s_{t})$ , so $r_{t} (θ)$ grows. without a limit, $r_{t} (θ)$ could become very large, causing a huge update that overshoots and harms performance.

ppo introduces clipping to prevent this:

L^{clip} (θ) = 𝔼_{t} [min (r_{t} (θ) A_{t}, clip (r_{t} (θ), 1 - ϵ, 1 + ϵ) A_{t})]

where $ϵ$ is a clipping parameter (like 0.2). the function $clip (r_{t} (θ), 1 - ϵ, 1 + ϵ)$ ensures $r_{t} (θ)$ stays between $1 - ϵ$ and $1 + ϵ$ (e.g., 0.8 to 1.2). if $A_{t} > 0$ , we increase $r_{t} (θ)$ , but cap it at $1 + ϵ$ ; if $A_{t} < 0$ , we decrease $r_{t} (θ)$ , but cap it at $1 - ϵ$ . the $min$ ensures we take the smaller of the two terms, limiting the update size.

to see why this works, consider the unclipped term $r_{t} (θ) A_{t}$ . if $A_{t} > 0$ and $r_{t} (θ)$ grows to, say, 10, the update would heavily favor $a_{t}$ , potentially ignoring other good actions. with clipping, if $r_{t} (θ) > 1 + ϵ$ , we use $(1 + ϵ) A_{t}$ instead, keeping the update bounded. this trust region approach makes ppo more stable than vanilla policy gradients.

but ppo still relies on a critic to compute $A_{t}$ , bringing back the same memory and accuracy issues. plus, for llms, generating samples and optimizing is already costly, and ppo’s critic and multiple epochs add to this burden. we need a method that avoids the critic, cuts compute costs, and maintains stability.

why grpo is the answer

let’s summarize the gaps in these methods and see how grpo fills them:

critic overhead: ppo needs a critic, doubling memory needs. grpo gets rid of the critic by estimating advantages directly from the policy’s outputs.
stability in llm training: llms have huge action spaces (e.g., a 50,000-token vocabulary), and big updates can lead to reward hacking—like generating gibberish that scores high. grpo uses clipping and a kl penalty to keep updates safe.
variance without a critic: without a critic, we risk high variance, as in vanilla policy gradients. grpo reduces variance by sampling multiple outputs and normalizing their rewards.
efficiency for reasoning: in llm reasoning tasks, like solving math problems, rewards are sparse—only the final answer matters. grpo’s group sampling fits this setup by comparing multiple answers to the same prompt.

imagine training an llm to solve math problems. instead of relying on a critic, you let the model generate several answers to the same problem, score them (like checking if the answer is correct), and use those scores to adjust the policy. grpo makes this process efficient and stable—let’s build it up.

building grpo: group sampling and advantages

for each prompt $q$ (the state $s$ , like a math question), grpo generates $G$ outputs ${a_{1}, a_{2}, \dots, a_{G}}$ using the current policy $π_{θ}$ . each output $a_{i}$ is a sequence of tokens—think of $a_{i}$ as one complete answer to the math problem, made up of tokens $a_{i, 1}, a_{i, 2}, \dots, a_{i, N}$ , where $N$ is the sequence length.

scoring the outputs

we need to evaluate each output to guide the policy. since $a_{i}$ is a sequence, we score it token by token, considering the partial sequence up to each position. for the $t$ -th token in sequence $a_{i}$ , we compute a reward that combines a raw score from a reward model with a penalty to prevent the policy from straying too far.

let’s define the raw reward first. we use a reward model $r_{ϕ} (q, a_{i, \leq t})$ , which scores the partial sequence $a_{i, \leq t} = (a_{i, 1}, a_{i, 2}, \dots, a_{i, t})$ given the prompt $q$ . this could be, for example, a measure of how correct the partial answer is at step $t$ . in a math problem, if the partial sequence is leading toward the correct solution, $r_{ϕ}$ might give a positive score; if it’s going off track, the score might be negative.

now, we don’t want the policy $π_{θ}$ to deviate too much from a reference policy $π_{ref}$ (often the initial supervised fine-tuned model), because large deviations can lead to reward hacking—where the model learns to exploit the reward model in unintended ways, like generating nonsense that somehow scores high. to control this, we add a penalty based on the kl divergence between $π_{θ}$ and $π_{ref}$ .

the kl divergence measures how much one probability distribution differs from another. for two policies in a given state, the kl divergence from $π_{ref}$ to $π_{θ}$ is:

KL (π_{ref} ∣ π_{θ}) = 𝔼_{π_{ref}} [\log \frac{π_{ref} (a ∣ s)}{π_{θ} (a ∣ s)}]

in our case, the state at step $t$ is the prompt $q$ plus the tokens generated so far, $a_{i, < t} = (a_{i, 1}, \dots, a_{i, t - 1})$ . we’re interested in the action $a_{i, t}$ , the $t$ -th token. the kl divergence for this step would be:

KL (π_{ref} ∣ π_{θ}) = \sum_{a} π_{ref} (a ∣ q, a_{i, < t}) \log \frac{π_{ref} (a ∣ q, a_{i, < t})}{π_{θ} (a ∣ q, a_{i, < t})}

we don’t compute the full kl divergence in practice, because summing over all possible actions $a$ (e.g., the entire vocabulary) is expensive. instead, we approximate its effect by penalizing the log ratio for the action $a_{i, t}$ that was actually taken:

\log \frac{π_{θ} (a_{i, t} ∣ q, a_{i, < t})}{π_{ref} (a_{i, t} ∣ q, a_{i, < t})}

this is the log of the ratio of probabilities under the two policies. if $π_{θ} (a_{i, t} ∣ q, a_{i, < t}) > π_{ref} (a_{i, t} ∣ q, a_{i, < t})$ , the ratio is greater than 1, and the log is positive; if $π_{θ}$ assigns a lower probability, the log is negative. we subtract this term, scaled by a coefficient $β$ (like 0.1), to penalize deviations from $π_{ref}$ . so the per-token reward is:

r_{i, t} = r_{ϕ} (q, a_{i, \leq t}) - β \log \frac{π_{θ} (a_{i, t} ∣ q, a_{i, < t})}{π_{ref} (a_{i, t} ∣ q, a_{i, < t})}

if $π_{θ}$ makes $a_{i, t}$ much more likely than $π_{ref}$ , the log term is positive, and the penalty reduces the reward, discouraging large deviations. if $π_{θ}$ makes $a_{i, t}$ less likely, the log term is negative, increasing the reward, but $β$ ensures this effect is controlled.

to get the total reward for the sequence $a_{i}$ , we sum over all tokens:

r_{i} = \sum_{t = 1}^{N} r_{i, t}

this gives us a reward $r_{i}$ for each output $a_{i}$ , balancing the quality of the sequence (via $r_{ϕ}$ ) with a constraint on policy deviation (via the kl penalty).

computing advantages

now we have rewards ${r_{1}, r_{2}, \dots, r_{G}}$ for the $G$ outputs. we need to compute advantages to guide the policy update, but without a critic. grpo’s insight is to use the group of outputs to create a relative measure of performance.

imagine the $G$ outputs as different attempts at solving the same math problem. some answers might be correct (high reward), while others are wrong (low reward). we want to encourage the policy to favor the better attempts, but we need a baseline to compare against. instead of using a critic, grpo uses the group itself as the baseline by normalizing the rewards:

{\hat{A}}_{i, t} = \frac{r_{i} - mean (r)}{std (r)}

where $mean (r)$ is the average reward across the group:

mean (r) = \frac{1}{G} \sum_{i = 1}^{G} r_{i}

and $std (r)$ is the standard deviation:

std (r) = \sqrt{\frac{1}{G} \sum_{i = 1}^{G} (r_{i} - mean (r))^{2}}

let’s see why this normalization helps. without normalization, we’d use $r_{i}$ directly, but the raw rewards can vary widely depending on the task. for example, if rewards range from -10 to 10, the updates would be noisy, similar to the variance issue in vanilla policy gradients. by subtracting $mean (r)$ , we center the rewards around 0:

𝔼 [{\hat{A}}_{i, t}] = 𝔼 [\frac{r_{i} - mean (r)}{std (r)}] = \frac{𝔼 [r_{i}] - mean (r)}{std (r)} = 0

dividing by $std (r)$ scales the values so the variance becomes 1:

Var ({\hat{A}}_{i, t}) = Var (\frac{r_{i} - mean (r)}{std (r)}) = \frac{Var (r_{i})}{(std (r))^{2}} = 1

this gives us a standardized measure: ${\hat{A}}_{i, t} > 0$ means $a_{i}$ performed better than the group average, and ${\hat{A}}_{i, t} < 0$ means it performed worse. for simplicity, we use the same advantage for all tokens in sequence $a_{i}$ , though some implementations compute per-token advantages. this group-relative approach reduces variance without needing a critic, as the group provides the context for comparison.

the grpo objective

now we use these advantages to update the policy, ensuring the updates are safe. grpo adopts a clipped objective similar to ppo:

L^{GRPO} (θ) = 𝔼_{t} [min (r_{t} (θ) {\hat{A}}_{t}, clip (r_{t} (θ), 1 - ϵ, 1 + ϵ) {\hat{A}}_{t})]

where:

r_{t} (θ) = \frac{π_{θ} (a_{t} ∣ s_{t})}{π_{θ_{old}} (a_{t} ∣ s_{t})}

here, $s_{t}$ is the state at step $t$ , which in our case is $(q, a_{i, < t})$ , and $a_{t}$ is the token $a_{i, t}$ . the expectation is over all tokens in all sequences. the clipping with $ϵ$ (like 0.2) ensures we don’t change the policy too much, just like in ppo, while ${\hat{A}}_{t}$ guides the update based on the group-relative performance.

the grpo algorithm

let’s put it all together:

generate $G$ outputs ${a_{1}, a_{2}, \dots, a_{G}}$ for each prompt $q$ using $π_{θ}$ .
compute rewards $r_{i}$ for each output by summing per-token rewards with the kl penalty.
calculate advantages ${\hat{A}}_{i, t}$ by normalizing the rewards.
optimize $L^{GRPO} (θ)$ using gradient ascent over a few epochs.
set $π_{θ_{old}} \leftarrow π_{θ}$ , and repeat.

wrapping up

grpo is a clever solution for llm training, removing the need for a critic, reducing variance through group sampling, and ensuring stability with clipping and kl penalties. it’s especially well-suited for reasoning tasks, making rl training more efficient and practical. if you have questions or want to dig deeper, let me know!