Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

Source: arXiv:2606.20537 · Published 2026-06-18 · By Liang Su

TL;DR

This paper addresses a key bottleneck in low-latency, small-batch on-device serving of large language models (LLMs) and physical-AI workloads, where interactive sessions repeatedly branch, reset, and re-enter at tight responsiveness budgets. Existing KV-cache reuse approaches (paged and radix key-value caches) optimize for high-throughput, high-concurrency workloads and only manage positional token fragment state. In contrast, the authors introduce execution-state capsules, a graph-bound checkpoint and restore mechanism that snapshots the complete committed execution state—including KV cache, recurrent state, convolution state, multi-token prediction state (MTP), and metadata—into contiguous static GPU buffers. This enables fast, exact restore, fork, and rollback operations at meaningful token boundaries without recomputing prefixes. Their white-box kernel runtime backend, FlashRT, runs captured CUDA computation graphs over these buffers with no block-table indirection, making state freezing feasible and fast.

The results show on an NVIDIA RTX 5090 that capsule restore is byte-identical at stored-state level and token-identical under greedy decode, unlike a KV-only restore which diverges. The GPU-resident snapshot and restore are sub-millisecond, and warm start time-to-first-token (TTFT) speedups increase significantly with prefix length, up to 27× at 16k tokens over cold prefill. Similar correctness and speed gains are replicated on embedded and large systems like Jetson AGX Thor and DGX Spark. The work defines a complementary latency-first serving design point for small-batch, on-device interactive AI serving, focusing on explicit reuse of the entire execution boundary rather than just token fragment caches optimized for throughput under concurrency.

Key findings

• Capsule restore is byte-exact at the stored-state level and token-identical under greedy decode on an RTX 5090, verified for LLMs and vision-language-action policies.
• A positional KV-only restore diverges immediately, proving that recurrent state reuse is critical (Fig 7.2).
• Capsule snapshot and restore GPU-resident operations complete in under 1 millisecond.
• Time-to-first-token (TTFT) speedup over cold prefill grows from 3.9× at 2k tokens to 27× at 16k tokens on RTX 5090.
• On Jetson AGX Thor, cold prefill costs seconds, leading to cold-to-capsule speedups from 9× up to 76×.
• Static contiguous buffer design avoids block-table indirection, enabling a single replayable CUDA graph and making the whole execution state a freezable, self-contained object.
• The chunk-alignment condition (snapshot boundaries aligned to linear-attention chunk size) is required for exact token-level replay.
• Capsule 'fork' verb can restore one capsule into N independent sessions with token-exact equivalence.

Threat model

n/a — This work focuses on runtime execution state management for latency optimization and correctness in interactive serving, not on adversarial security threats or attacks.

Methodology — deep read

1. Threat model & assumptions: The paper assumes an adversarially benign environment focused on correctness and latency rather than security attacks. The primary goal is fast, low-latency serving for single or few concurrent streams on-device, under a fixed model and hardware setting. The adversary is essentially system-level variability or approximation that might break state equivalence.

2. Data: The evaluation uses a hybrid linear- and full-attention LLM model with prefixes up to 16k tokens, tested on NVIDIA RTX 5090, Jetson AGX Thor, and DGX Spark devices. The data consists of token sequences representing coding assistants and vision-language-action policies. Token streams and model states are labeled and verified for equivalence. No external datasets per se; rather focuses on serving traces.

3. Architecture / algorithm: FlashRT runtime captures the entire forward pass as a CUDA computation graph over static, contiguous GPU buffers (named buffers for KV cache, recurrent fold, convolution state, MTP state, metadata). Unlike prior works (vLLM, SGLang) that manage token-addressable KV fragments with indirection, FlashRT binds a fixed buffer set without block-table lookups, enabling a self-contained snapshot of the execution boundary. The execution-state capsule is defined as this frozen buffer set at a committed token boundary. Serving verbs (snapshot, restore, fork, rollback) manipulate these capsules by byte-copying the entire buffer set plus replaying the pre-captured CUDA graph with appended tokens.

4. Training regime: Not applicable—this is a serving/runtime infrastructure paper, not a model training contribution.

5. Evaluation protocol: The authors measure time-to-first-token (TTFT) latency across cold prefill baseline versus capsule warm start on three hardware platforms. They verify correctness at three layers: (1) byte-level restore: exact copying of stored GPU buffers; (2) state completeness: adversarial overwrite of buffers then restore yields token-identical greedy decoding; (3) token-level decode equivalence of restore+append versus cold prefill decode, including forked parallel sessions. Ablations show KV-only restore diverges immediately, validating the need for full execution-state reuse. Chunk alignment is tested as a necessary condition for exact equivalence. Statistical testing specifics are not described.

6. Reproducibility: The paper references an available GitHub repo for FlashRT (https://github.com/flashrt-project/FlashRT). The final evaluation is on closed hardware platforms and a production hybrid model; no publicly released trained models or datasets are described. Code release is partial and infrastructure-dependent.

Technical innovations

• Definition of execution-state capsules as graph-bound, contiguous static buffer snapshots capturing all necessary model execution state (KV, recurrent, convolution, MTP, metadata) at committed token boundaries.
• Design of FlashRT runtime as a latency-first single-stream CUDA Graph launcher over static buffers, avoiding block-table indirection to enable freeze/restore/fork/rollback operations on entire execution state.
• Formalization of a minimal execution contract exposing three handle types and an opaque shape key to realize cross-space asynchronous snapshot and restore within the capsule mechanism.
• Identification of chunk-alignment condition whereby snapshot boundaries must respect linear-attention chunk sizes to guarantee exact token-level replay equivalence.

Baselines vs proposed

• vLLM cold prefill TTFT (2k tokens) = baseline, FlashRT capsule TTFT = 3.9× faster
• vLLM cold prefill TTFT (16k tokens) = baseline, FlashRT capsule TTFT = 27× faster
• vLLM automatic prefix caching warm reuse TTFT speedup ≈ 1.4–2.8× vs FlashRT capsule greater speedup
• Jetson AGX Thor cold prefill TTFT costs seconds, capsule yield 9–76× speedup
• KV-only restore diverges token output immediately vs full capsule restore token-identical

Limitations

• The evaluation focuses on low-latency single/few-stream on-device serving, not high-throughput or distributed serving scenarios.
• Correctness checks cover token-level identity under greedy decode but do not test robustness to adversarial inputs or model updates.
• The chunk alignment condition requires pinned snapshot boundaries, which may complicate some interaction patterns.
• Steady-state decode throughput and speculative decoding improvements are out of scope and not evaluated.
• No detailed evaluation of memory overhead or impact on large model VRAM budgets beyond qualitative discussion.
• Production multi-turn agent serving and on-robot field evaluations are future work.

Open questions / follow-ons

• How to extend execution-state capsule concepts to support high-concurrency, multi-client serving scenarios with shared prefix reuse?
• Can capsule mechanisms be integrated with dynamic batching or adaptive precision serving to further optimize latency?
• What policy strategies for capsule pinning, eviction, and tiering maximize VRAM efficiency under constrained on-device memory?
• How do capsules perform under models with more complex recurrent states or attention mechanisms beyond linear attention?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on real-time interaction systems employing LLMs or embodied agents, execution-state capsules offer a novel mechanism to minimize end-to-end latency on-device by reusing the entire execution state boundary rather than just token-level KV caches. This mechanism supports rapid session warm starts, branching, and rollbacks critical when user interactions require immediate responsiveness, interruptions, or multi-path exploration (e.g., tree-of-thought). Unlike throughput-optimized KV cache reuse that suits multi-tenant high concurrency, capsules enable latency-first serving on constrained edge or embedded GPUs where responsiveness matters most. Understanding this design allows bot-defense engineers to architect systems that handle user session context efficiently while maintaining correctness, an important consideration for CAPTCHA challenges with adaptive interactions or multi-turn AI assistants embedded in client devices.

Cite

@article{arxiv2606_20537,
  title={ Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving },
  author={ Liang Su },
  journal={arXiv preprint arXiv:2606.20537},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20537}
}

Read the full paper

• arXiv:2606.20537 (abstract)
• arXiv:2606.20537 (PDF)