Key modifications PPO/GRPO need on agents (must memorize): the token mask must restrict the loss to tokens the agent itself generates — observation tokens (the stdout / search snippet returned by ...