RL Post-Training Stalls at 256K Tokens. LongStraw Runs at 2.1M.

The field has quietly accepted a strange asymmetry: inference systems run at million-token contexts while RL post-training stalls at 256K tokens or below, relying on length generalization to cover the gap at deployment. For AI agents, that gap is not academic. Observations accumulate, tool outputs stack, prior decisions persist, and by the time a trajectory reaches the point where context length actually matters, the training regime that shaped the model never saw anything like it.

LongStraw is an architecture-aware execution stack built specifically to close that gap under a fixed GPU budget, instantiated with Group Relative Policy Optimization (GRPO). The core insight is that not every part of a long-context RL forward pass needs to live in the training graph simultaneously. The shared prompt, which can stretch across millions of tokens, gets evaluated without autograd. Only the model-specific state needed by later tokens is retained. Response branches, which are short relative to the full context, are replayed one at a time rather than batched in memory. The result is a dramatic reduction in the live training graph, traded against additional replay time. Think of it as a checkpoint-and-replay strategy applied not to gradient accumulation but to the structure of the RL rollout itself: pay compute cost, not memory cost.

The architecture targets two distinct model families, which matters because the memory profile of long-context RL differs by attention type. For Qwen3.6-27B, a hybrid recurrent and full-attention model, and for GLM-5.2, a compressed-attention mixture-of-experts model, LongStraw implements separate execution paths that respect each architecture's internal state structure. Scaling group size in GRPO, normally a significant memory multiplier, adds only 0.21 GB of peak allocated memory when going from a group of 2 to a group of 8 at 2.1M positions. That number is the key signal: group size, which controls the quality of the relative reward signal in GRPO, is no longer a memory bottleneck.

On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions. A separate stress test reaches 4.46M positions. On 32 H20 GPUs, the end-to-end execution path validates across all 78 layers of GLM-5.2 at a 2.1M-token prompt. One caveat belongs here and should not be buried: these experiments establish execution capacity, not full training correctness. The captured prompt state is detached, and some distributed forward and gradient composition paths remain incomplete. For teams evaluating adoption, this is a proof of execution at scale, not a drop-in training recipe. For ML infrastructure teams building toward agent post-training at deployment-realistic context lengths, the takeaway is direct: the hardware constraint on long-context RL is not the binding limit, and the path to closing the gap runs through algorithmic execution design, not GPU procurement.

We're thinking: The 256K ceiling on RL post-training has been treated as a hardware problem, which means labs have been waiting for better GPUs rather than questioning the execution architecture. LongStraw shows the ceiling is algorithmic: the same eight H20s that couldn't run GRPO at 2.1M tokens can do it once the forward pass is restructured to separate prompt evaluation from gradient tracking. We find that implication more significant than the token count itself. If the bottleneck was always in execution design, then the capability gap between how agents are trained and how they are deployed has been a choice, not a constraint. The incomplete gradient paths are a real limitation today, but the existence proof at 2.1M and 4.46M positions shifts the conversation from "when will hardware catch up" to "how quickly can execution stacks mature."

Key takeaways:

• LongStraw separates prompt evaluation from the live training graph, replaying short response branches individually to trade compute time for memory, enabling GRPO at context lengths previously blocked by GPU memory limits.
• On eight H20 GPUs, grouped scoring and response backward complete at 2.1M positions with group size scaling adding only 0.21 GB of peak memory; a stress test reaches 4.46M positions, though full training correctness is not yet validated due to detached prompt state and incomplete gradient composition paths.
• Teams building RL post-training pipelines for long-context agents should treat LongStraw as a reference architecture for execution design, audit their own prompt-versus-response memory split, and track the gradient composition work before committing to production adoption.

Source: LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget