GRPO Reward Shaping for Language Models

Reward shaping for language models is the central design problem that Group Relative Policy Optimization (GRPO) leaves to practitioners. Removing the critic simplifies training, but it does not resolve how useful behavior earns credit. The policy learns the intended behavior or exploits the signal supplied to it according to how rewards are constructed, calibrated, and decomposed.

What GRPO actually does and what it leaves unresolved

GRPO samples a group of responses for each prompt, typically 8–64, and scores each response. It then subtracts the group mean reward from each response’s reward and divides by the group standard deviation. This normalized advantage replaces the baseline supplied by a learned value function. There is no critic error to compound. Eliminating that model makes training more stable and memory-efficient than PPO, while a KL penalty against a reference policy and a clipped policy ratio remain guardrails against drift.

Those guardrails constrain updates, but they do not identify which reasoning steps deserve credit. A critic supplied a baseline that helped decompose credit across the trajectory. Without that decomposition, every token in a response inherits the same scalar signal. When a correct answer comes from a flawed intermediate step, that step gets rewarded along with the useful ones. Conversely, sound intermediate work in an unsuccessful response receives the same unfavorable signal as the error that defeats it.

The group baseline also depends on differences among responses. When the policy converges on a single reasoning pattern, additional rollouts become redundant. Coarse credit assignment and diversity collapse are predictable consequences of relying on group-relative rewards without a learned critic, and they define the problems that reward shaping must address.

Why the reward function is now the primary lever practitioners control

Within this critic-free update, the reward function is where practitioner judgment about desired behavior enters the optimization signal. Binary correctness leaves much of that judgment unstated. It cannot distinguish a direct solution from a roundabout one, assess the quality of intermediate reasoning, or recognize when a chain has already reached its conclusion. Both paths can earn the same reward. On hard prompts, sparse correctness rewards can also leave the group without a useful gradient.

Auxiliary terms supply distinctions the task reward omits, but they create a calibration problem. A length reward can discourage verbosity while becoming strong enough to compete with correctness. A diversity reward can preserve exploration while rewarding variation that adds no value. The difficulty is expressing the relative priority of desirable properties so that optimizing a supplement does not displace the task.

The diagnostic distinction is between how trajectories are sampled and how rewards are constructed. These are the two axes of a policy-gradient objective, and GRPO concentrates much of the available design freedom on the reward side. Reward hacking, verbosity bias, mode collapse, and difficulty insensitivity often reveal how that signal works. Yet sampling determines which behaviors the reward can compare. Reward shaping, curriculum design, and advantage reweighting consequently operate as coupled choices.

Token-level reward redistribution as a structural response to coarse credit assignment

The model’s uncertainty offers a way to distinguish decisions within a response without adding labeled data. At each generation step, the probability distribution over possible tokens has a Shannon entropy. High entropy indicates uncertainty and correlates with pivotal reasoning junctures. These are decisions where credit has particular leverage. In successful sequences, uncertain choices get bonuses because exploration led to a correct outcome, while confident choices in unsuccessful sequences get larger penalties.

Group Token Policy Optimization (GTPO) implements this redistribution at the token level. Sequence-Level GRPO (GRPO-S) applies the same logic using average sequence entropy, trading fine-grained allocation for computational efficiency. The latter is more stable in long chain-of-thought tasks. The choice therefore concerns the resolution of credit assignment within the same task objective. Both use uncertainty information already computed during generation, so they need no separate reward model or change to the rollout procedure.

The redistribution also has constraints that keep it grounded. When weighting coefficients sum to one, the total shaped positive reward across successful sequences equals their total original positive reward at every timestep. GTPO’s asymptotic consistency result addresses the further concern that entropy weighting introduces lasting bias. As entropy among successful sequences diminishes during training, the gradient bias vanishes, and GTPO converges to the same optimum as the baseline.

Diversity-aware reweighting and the exploitation trap in group rollouts

More precise credit within a response does not make a redundant group informative. Once a policy begins converging, it can generate near-identical completions for the same prompt, leaving each rollout compared with copies of itself. The effective group size shrinks without reducing computation. Its mean becomes a less useful reference for comparing behavior.

MMR-GRPO, published in Findings of ACL 2026, addresses this exploitation trap through Maximal Marginal Relevance. It reweights rollout rewards by balancing quality against similarity to already-selected rollouts. Distinct, high-quality responses receive higher effective reward weight, while redundant responses are downweighted even when correct. Quality remains the primary criterion. Diversity acts as a tiebreaker among similarly scoring rollouts, so the objective is not variation at the expense of correct behavior.

MMR-GRPO reached about the same peak performance as standard GRPO across multiple model sizes and five benchmarks, using far fewer training steps and less wall-clock time. That finding gives the diversity mechanism a practical basis. Its efficiency comes from making better use of each rollout.

Gradient vanishing and diversity collapse as trajectory-side problems that reward shaping can partly address

When rewards are the same across the group, reweighting cannot get past the limit of group normalization. If every rollout succeeds on an easy prompt, all receive the same correctness reward. If every rollout fails on a hard prompt, the result is equally uniform. In either case, the advantages collapse to zero. There is no relative learning signal to amplify.

TA-GRPO changes the trajectories available for comparison by automatically generating semantically equivalent rephrasings of each training question. The wording, format, and order of information change, but the problem stays the same. These changes can make the question seem harder or easier and lead to different reasoning strategies. Advantages are computed jointly over responses to the original question and its rephrasings, allowing the expanded group to supply variance absent from the original group. Importance sampling ratios remain anchored to the original question to keep the update semantically grounded.

This qualifies the claim that reward shaping is the primary lever. Auxiliary terms cannot help if they leave every response identically rewarded, and multiplying zero advantages cannot recover a signal. The missing distinction may require changing what the model responds to rather than changing how existing responses are scored. Gradient vanishing and diversity collapse can be coupled failures that require joint design of the trajectory and reward sides.

Shapley decomposition for set-level tasks where individual candidate rewards are undefined

Set-completion tasks pose a different problem because the reward may belong to a collection. In recommendation or brainstorming, utility can depend on topic coverage, diversity, or the joint quality of the candidates. An item that is excellent on its own may add only what another item already adds. Its value depends on its companions. Assigning the aggregate reward uniformly therefore credits redundant items for utility they did not add.

A Shapley value measures an item’s marginal contribution averaged across all orderings of the set’s members. ShapE-GRPO uses this construction from cooperative-game-theory to turn the utility of the collection into rewards for each candidate. The decomposition satisfies efficiency, meaning contributions sum to the total reward. It also satisfies symmetry, giving identical contributors identical credit, and the null-player property, giving zero credit to candidates that contribute nothing.

This differs from downweighting similar rollouts in an ordinary response group. Here, individual utility is not available until the set-level reward has been decomposed. The decomposition supplies the missing unit of credit. Its naive computation is exponential in the number of candidates, although approximations exist. The architectural gain is that a collective objective becomes a per-candidate signal that GRPO’s advantage computation can use without treating every member as equally responsible for the outcome.

Length and difficulty shaping as calibration tools for the primary reward signal

For calibration, the governing principle is to distinguish behavior within a correctness class without overriding correctness itself. Consider length measured only among correct rollouts. Z-scoring those lengths gives an auxiliary reward a relative basis for penalizing verbosity, rather than allowing brevity to redeem an incorrect answer. GRPO-LEAD applies an exponential decay function to that standardized measure. Correct responses remain positively rewarded, while incorrect completions receive a fixed negative reward that preserves the boundary between success and failure.

GRPO-LEAD also uses each group’s correctness ratio as a proxy for difficulty. A logistic weighting function amplifies advantage signals for harder prompts, adjusting their influence without changing the trajectories. This targets weak learning signals where responses differ in useful ways. It does not remove the identical-reward limit described earlier. Difficulty reweighting can strengthen an available gradient, but it cannot manufacture one from uniformly zero advantages.

LASER takes a threshold-based approach to length calibration. Correct responses within a target length get an auxiliary bonus, while those outside it get none even if correct. The bonus cannot make an incorrect response positive. LASER-D adapts the target to both prompt difficulty and the evolving policy, avoiding a uniform length constraint across problems of different complexity. The resulting optimal balance favors shorter reasoning where shorter reasoning is achievable, rather than treating compression as an unconditional objective.

These designs make the practical guidance precise. Auxiliary rewards should supplement the primary objective through their structure. A correctness-gated bonus limits what length optimization can accomplish, while within-class normalization gives its magnitude a relevant comparison. Calibration is therefore more than tuning reward weights. It determines which trade-offs the optimization is permitted to make.

Reward hacking as the failure mode that reward shaping introduces and must guard against

Every auxiliary reward gives the policy another property to optimize on its own, apart from the intended outcome. A length threshold can favor responses that qualify as short without genuine efficiency. A diversity reward can favor surface differences between semantically equivalent answers, while format or style rewards can favor following the rubric over doing the task well. Optimization follows the supplied signal regardless of the unstated intention. Whenever the two diverge, the shaped reward offers an exploitable route.

Structural defenses make that divergence harder to sustain. Multiple independent reward models can require agreement, reducing reliance on a single model that the policy may learn to game. Adversarial reward loops make the evaluator adaptive. Adv-GRPO uses this architecture for adversarial reward design in generated images, jointly training a discriminator to distinguish them from high-quality reference images and using its signal as reward.

Held-out behavioral tests add another way to check quality. They define invariants the policy must satisfy beyond the shaped training signal, with failures treated as evidence of reward hacking. For writing systems, those invariants can cover how well the writing keeps to the voice, follows style constraints, and stays grounded in facts. The same logic that tests software can therefore test whether rewarded writing behavior preserves the actual objective. A writing kernel built on GRPO-family optimization can make this discipline explicit and automatable rather than relying solely on post-hoc review of every output.

The governance consequence is that rewards are specifications. They need versioning, review, and testing with the same rigor as the weights they shape. Token redistribution, rollout reweighting, and set decomposition each encode a judgment about what deserves credit. Changing any of them changes the behavior the system is being asked to learn. Reward shaping becomes a reliable way to design training only when those judgments stay open to inspection and adversarial review.