Together AI open-sources ThunderAgent agentic inference layer
ThunderAgent schedules whole agent workflows instead of single calls, reporting 2x throughput and 6x lower latency than SGLang on one 8xH100 node.
Together AI open-sourced ThunderAgent on Wednesday, a scheduling layer that sits between agent clients and inference engines and treats an entire multi-turn agent workflow — not an individual model call — as the unit of scheduling.
The problem it targets is KV cache thrashing. When an agent pauses to run a tool call, a conventional scheduler sees an idle request and evicts its cached attention state to serve other traffic; when the agent resumes, that state has to be recomputed. At scale, with thousands of concurrent agent programs each pausing repeatedly, the recomputation dominates. ThunderAgent asks clients to attach a single program_id field to their requests, which is enough for the scheduler to know which calls belong to the same workflow and to keep their caches coherent. Under memory pressure it pauses whole low-priority programs rather than shredding everyone's cache, so the surviving workflows hit far higher cache-hit rates.
The reported numbers: on a single 8xH100 node at batch size 192, ThunderAgent sustained 803 tokens per second at 10.6 seconds mean latency, against SGLang's 390 tokens per second at 65 seconds — roughly double the throughput at a fraction of the latency. Multi-node, throughput scaled near-linearly from 671 to 2,248 steps per minute going from 16 to 64 GPUs, and the margin over SGLang's gateway widened with cluster size, from 1.79x at two nodes to 2.39x at eight. Together also cites gains of 1.8x to 3.9x on reinforcement-learning rollout workloads, where the same pause-and-resume pattern appears during synthetic data generation.
The project is positioned as a drop-in: it is format-agnostic and composes with existing engine features such as KV offloading and speculative decoding rather than replacing them. Code is on GitHub, and the accompanying paper was accepted to ICML 2026 as a spotlight.
Why it matters: Serving infrastructure was designed for chat — short, self-contained requests — while the fastest-growing traffic is long agent programs that idle constantly. Reclaiming a 2x throughput factor at the scheduler layer, with no model change and one added field, cuts the cost of running agents in production immediately, and it does so in the open rather than as a proprietary advantage for one inference vendor.