Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO

New empirical follow-up: Part 2 compares seven trainer configurations on Qwen3-14B and our DEX gym, with no-think and thinking holdouts, interactive reward curves, and downloadable data. It is a separate experiment from the 27B table below and does not establish a universal trainer ranking.

If you only ever train language models on toy math puzzles, reinforcement learning feels simple: did the model output 42? If yes, reward is 1. If no, reward is 0.

The moment you try to train an autonomous agent for real-world enterprise work, however, a single scalar reward is an absolute illusion.

In production, your agent has to juggle multiple competing, messy, non-commensurate priorities at the same time:

• The Main Mission (R₁): Did it actually solve the customer’s request? (e.g., execute the right SQL query, compile the circuit, return the right data payload).
• Execution Efficiency (R₂): Did it solve it elegantly in 3 tool calls, or did it run a wild 40-step loop that burned $4 in API tokens and spiked database CPU?
• Hard Guardrails & Constraints (R₃): Did it stay inside the sandbox? Did it strictly adhere to output JSON schemas, avoid mutating production tables, and preserve security invariants?

Here is the dirty secret of post-training: if you take these three scores and simply add them together into standard algorithms like PPO or vanilla GRPO, your training run will almost certainly tear itself apart. The loudest reward channel swallows the subtle ones, the agent learns to game the system, and up to a third of your expensive GPU batches end up generating zero gradients.

Start here

Understand how several distinct reward channels are combined into policy advantages, why simple weighted sums cause catastrophic scale dominance, how GDPO decouples channel normalization, and why hard constraints require enforcement rather than reward weights.

• Policy: The neural network that chooses the next token or action given the current context.
• Rollout: One complete candidate attempt at a task, including intermediate tool calls and final answers.
• Reward channel: An independent score assigned to an attempt by a specific evaluator (e.g. task success, latency, format).
• Advantage: A normalized scalar indicating how much better or worse an attempt was compared to its baseline.
• Scale dominance: When a high-variance reward channel mathematically drowns out subtle constraint metrics during joint normalization.

1. The Algorithmic Evolution: PPO → GRPO → DAPO → GDPO

To understand why modern multi-reward agent post-training looks the way it does, we must trace how policy gradient estimators evolved:

1.1 PPO: The VRAM-Hungry Workhorse

Proximal Policy Optimization (Schulman et al., 2017) was the engine behind the original RLHF revolution. It uses an Actor-Critic architecture: the Actor generates the tokens, and a separate Critic network learns to predict the expected future reward from state s.

Why PPO Hurts in Practice:

1. The VRAM Double-Tax: If your policy is a 27B model, your Critic is usually another 27B model. You have to hold two massive models in GPU memory along with their optimizer states and activations. You end up needing twice as many H100s just to keep the critic alive.
2. Critic Drift on Multi-Reward Tasks: Trying to train a single critic head to predict a composite stew of task accuracy, latency penalties, and format compliance is notoriously unstable. The critic gets confused, advantages get noisy, and policy updates turn sluggish.

1.2 GRPO: Ditching the Critic Entirely

DeepSeekMath (2024) introduced Group Relative Policy Optimization (GRPO), and it felt like a breath of fresh air. GRPO tossed the Critic network into the recycling bin.

Instead of asking a neural net to predict a baseline, GRPO samples a group of G candidate completions {o₁, o₂, ..., o_G} for the same prompt, scores them all, and normalizes advantages against the group’s own mean and standard deviation:

Instant win: GPU memory needs dropped in half! But when applied to multi-objective environments, vanilla GRPO made an innocent-looking mathematical assumption called Sum-then-Normalize:

As we will demonstrate below, this single line of math creates a devastating failure mode: Scale Dominance.

1.3 DAPO: Rescuing Dead Groups with Dynamic Sampling

When you run GRPO on difficult engineering problems, you quickly discover the curse of Dead Groups. If a coding task is tough and all 8 candidate rollouts in a group fail with a syntax error, every single rollout gets a reward of 0.

When all rewards are 0, the group standard deviation is 0. That means the advantage is 0 across the entire group! Your expensive GPU cluster just spent 30 seconds generating tokens, and the gradient update is completely empty. In hard tasks, 30% to 40% of all training steps can be dead groups.

DAPO introduced dynamic sampling: during rollout scoring, if a group has zero variance, it immediately discards the dead data and pulls fresh active prompts until every training batch contains real learning signal, cutting wasted GPU cycles to near zero.

A group where all 8 attempts fail has zero variance and gives no gradient. DAPO's dynamic sampling replaces it with a prompt whose attempts differ.

1.4 GDPO: Decoupled Normalization (Normalize-then-Sum)

Introduced in 2026 (arXiv:2601.05242) and integrated into modern libraries like TRL 1.7, GDPO fixes the fundamental multi-reward flaw of GRPO. Instead of adding raw scores together and then normalizing, GDPO enforces Normalize-then-Sum:

Every reward channel is normalized independently across the group first. Now, whether a reward channel naturally varies between [0, 1.0] or between [0, 0.05], both channels have a mean of 0 and a variance of 1. The small metric can no longer be bullied by the large one.

Sum-then-normalize (GRPO) lets the high-variance task reward drown out token efficiency; normalize-then-sum (GDPO) puts both channels on the same scale before weighting.

2. Worked Example: Does Changing Score Units Change the Learning Signal?

Suppose attempts A and B are correct, while C and D are wrong. Each also receives an efficiency score: B is both correct and efficient. C is wrong but very fast—returning an incorrect constant can be lightning-fast.

The figure below compares Sum, then normalize with Normalize each, then sum, and reports efficiency on a 0–1 and on a 0–100 scale:

One prompt, four attempts, equal weights. Summing raw scores lets a change of units (efficiency × 100) hand the top advantage to a wrong answer; normalizing each channel first does not depend on units.

The original post has an interactive version of this example.

Notice what happens under joint summation: when efficiency is reported on a 0–100 scale, the incorrect but fast candidate C suddenly receives a higher positive advantage than the correct solution! Under decoupled normalization, candidate B remains the standout winner regardless of unit scales.

3. The Mathematics of Scale Dominance: Why Simple Sums Fail

To see why Sum-then-Normalize fails mathematically, look at the variance of a sum of two independent reward signals:

In an agent task:

• Channel 1 (Task Success) is binary: r₁ ∈ {0, 1}. Its variance is roughly σ₁² ≈ 0.25.
• Channel 2 (Token Efficiency) is small: r₂ ∈ [0, 0.05]. Its variance is tiny: σ₂² ≈ 0.0006.

When you sum them up, Channel 1 accounts for 99.7% of the total variance. When GRPO divides by σ_R, Channel 2 is effectively multiplied by zero. The model quickly learns a toxic heuristic: “I can spam 50 unnecessary tool calls and burn huge token budgets, because the efficiency penalty is mathematically invisible to my gradients!”

4. Geometric Reward Collapse: When Winners Look Like Losers

Under vanilla GRPO, distinct trade-offs get mashed into the exact same scalar score:

• Candidate A: Perfect task solution (R₁ = 1.0), but horribly violates formatting rules (R₃ = 0.0). Total = 1.0.
• Candidate B: Flawless format and safe execution (R₃ = 0.2), but solved 80% of the core task (R₁ = 0.8). Total = 1.0.

To vanilla GRPO, both candidates look identical (A_A = A_B). The policy receives zero gradient to choose the clean, compliant solution over the broken, unsafe one. With GDPO’s decoupled normalization, Candidate B’s excellence on the constraint channel stands out with a strong positive sub-advantage, guiding the model toward the true Pareto frontier.

5. Hard Telemetry from gft-studio

We tested all four algorithms on an identical multi-objective agent gym in gft-studio, evaluating task success (R₁), token efficiency (R₂), and constraint compliance (R₃) on a 27B model:

The Big Takeaways

• Vanilla GRPO Bleeds Safety Constraints: While it solved the primary task 51.2% of the time, its constraint violation rate was alarming: 29.4%. Because task reward dominated the sum, the agent routinely broke formatting and safety contracts to get the job done.
• DAPO Saves 34% of Wasted Compute: Over a third of vanilla GRPO groups had zero variance. DAPO’s dynamic prompt replenishment ensured that every batch drove meaningful parameter updates.
• GDPO Nails the Pareto Frontier: By standardizing each reward channel independently, constraint violations collapsed from 29.4% down to under 3.8%, while overall task success jumped to 63.4%.

6. The Practitioner’s Field Guide

When building reinforcement learning pipelines for real-world enterprise agents:

1. Never use joint summation for multi-reward environments. Always normalize channels independently first (Normalize-then-Sum).
2. Hard constraints belong in the tool interface, not in soft weights. If an agent must never write unauthorized data to production, block the write at the API barrier. Do not rely solely on negative rewards to enforce safety.
3. Turn on dynamic group sampling (DAPO) if your task is hard. If your baseline model solves the problem less than 20% of the time, you will waste huge amounts of GPU compute on dead groups without it.
4. Clamp your combined advantages. When combining multiple normalized channels, an outlier rollout that spikes on two channels simultaneously can cause massive policy drift. Always apply a safety clamp (c_max ∈ [3.0, 5.0]) to protect training stability.

Originally published at g-ftech.com.