TraceLab: Characterizing Coding Agent Workloads for LLM Serving

Source: arXiv:2606.30560 · Published 2026-06-29 · By Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy et al.

TL;DR

The paper addresses the challenge of efficiently serving coding agents powered by large language models (LLMs), a fast-growing application area for agentic LLMs that engage in multi-step reasoning and tool invocation. Existing public datasets and benchmarks lack representative traces capturing the complex, multi-step, multi-tool workflows typical of real coding-agent usage. To fill this gap, the authors collect and release a large-scale dataset of approximately 4,300 real-world coding-agent sessions composed of 350,000 LLM steps and 430,000 tool calls, covering usage from 43 developers over about eight months on two major coding agents, Anthropic Claude Code and OpenAI Codex, spanning over 20 model versions.

Their detailed workload characterization reveals coding-agent sessions are largely autonomous with long agentic loops composed of tool-initiated LLM steps. Context lengths grow very large (median prefix context ~126k tokens per step), but each step appends relatively few new tokens (<1k medians). Outputs per step are short compared to the long input context, reflecting the multi-step tool interaction workflow. Tool calls are heavily skewed toward a small set, but show diverse latencies from milliseconds to hours. They find that despite caching mechanisms, prefix cache reads dominate serving costs. Human idle time dominates session wall-clock latency, but within requests, tool execution and generation have comparable latency shares. These detailed insights expose several critical system bottlenecks and propose concrete optimizations for serving coding-agent workloads.

The released dataset and associated analysis pipeline provide a first cross-provider, large-scale resource to study coding-agent workloads in production-like settings. This work advances understanding of the unique characteristics of coding-agent LLM workloads and informs future large-scale serving system designs and optimizations.

Key findings

• Collected 4,300 coding-agent sessions from 43 developers spanning Claude Code and Codex, totaling roughly 350,000 LLM steps and 430,000 tool calls.
• Sessions average 9.2 requests and 8.9 user-initiated plus 73.6 tool-initiated LLM steps, highlighting a mostly autonomous agentic loop.
• Median step reads back 126k prefix tokens but only appends 857 new tokens for Claude; Codex shows similar pattern with slightly shorter context.
• Outputs per step are comparatively short: median 252 tokens for Claude, 184 for Codex, reflecting fine-grained multi-step tool calls.
• Top 3 most popular tools account for 80%+ of calls; 4% of tool calls exceed 1 minute latency but consume 85% of total tool-call time.
• Prefix cache hit rate is high at 95.7%, yet misses cause 3.8x more tokens to be re-prefilled, making cache reads the cost dominant factor (59.5% of token costs).
• Average session costs $9.70 but median is $0.61 with heavy tail (p99 $178) mostly from long sessions; prefix tokens dominate cost despite cheaper per-token rate.
• Human wait time dominates session wall-clock at 92.3%, but within a request tool execution latency contributes 59.8% of response time versus 41.0% decoding.

Threat model

n/a - The paper does not focus on adversarial threats or attacks but rather characterizes benign real-world workloads from trusted developer users of coding agents to inform efficient serving.

Methodology — deep read

The authors begin with the threat model of serving real-world coding-agent workloads, considering typical users who interact with coding agents like Claude Code and Codex, and the challenges of long-session context management, multi-step LLM calls, and frequent tool invocations. No explicit attacker model is discussed, as the paper focuses on workload characterization for serving optimization.

They collected raw session logs from production-grade coding agents operated in their own research teams over roughly eight months. The data covers 4,265 sessions from 43 developers, totaling about 350,000 LLM invocations and 430,000 tool calls. The datasets include 23 model versions across the two providers. Each log contains timestamped events of user messages, LLM reasoning and output, tool calls, and tool results with token counts and timing metadata.

Due to schema differences between Claude Code and Codex logs, data normalization was performed into a unified step-level schema where each step includes one LLM call and associated tool calls. Token accounting was split into prefix tokens (cached history replayed), append tokens (new incremental input), and output tokens (generation). Timing metrics were extracted per step to analyze latencies of generation and tool execution.

Anonymization was applied by pseudonymizing user, session, and tool identifiers and removing raw user and tool message text to protect privacy. The resulting sanitized dataset preserves token and timing statistics necessary for system analysis.

The analysis pipeline computed per-session, per-request, and per-step distributions of step counts, context size changes, token counts, output lengths, latency breakdowns, and cost estimations using public API pricing for the models. Multiple detailed metrics characterize context growth and compaction behaviors, output token accounting schemes (output cached vs output resend), prefix cache hit rates, and tool call distributions by frequency and latency.

One concrete example: a typical request involves around 8 LLM steps and 11 tool calls. Each step has a median of about 126k prefix tokens and 857 append tokens, with output around 252 tokens. Most steps grow context size incrementally, but Codex also shows micro reductions. The serving cost breakdown highlights that prefix cache reads dominate spend despite the lower per-token cost. Latency analysis reveals that human delays dominate total session wall-clock but tool execution dominates request response time.

The study included cross-provider comparisons and temporal analysis across model versions. No experiments on adversarial behaviors were conducted. Reproducibility is supported by public release of the dataset, collection pipeline, and analysis code.

Technical innovations

• First large-scale cross-provider trace capturing real day-to-day coding-agent usage with detailed multi-step tool calling across diverse model versions.
• Unified normalization pipeline that harmonizes disparate raw logs from Claude Code and Codex into a consistent step-level schema enabling integrated analysis.
• Fine-grained decomposition of token accounting into prefix tokens (cached), append tokens (new), and output tokens to analyze context growth and cost contributions.
• Empirical analysis revealing agentic coding workflows are dominated by long autonomous loops with infrequent but impactful context compactions and diverse tool latency distributions.
• Identification and classification of output token reuse schemes (output cached vs output resend) informing KV-cache management optimizations.

Datasets

• TraceLab coding-agent trace — ~4,300 sessions, 350K LLM steps, 430K tool calls — collected from internal day-to-day usage of Claude Code and Codex across 43 developers over eight months, anonymized and public at https://github.com/uw-syfi/TraceLab.git

Baselines vs proposed

• No direct modeling baselines; cost and latency breakdowns compared between Claude Code and Codex models showing similar workload patterns but different context compaction and output-caching behaviors.
• Observed normalized decode speed median: Claude 46.8 tokens/s, Codex 33.9 tokens/s with high variance.
• Global prefix cache hit rate of 95.7%, with Codex gpt-5.4 showing mostly output-cached scheme and gpt-5.5 output-resend scheme.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30560.

Fig 1: Per-step prefix tokens vs. append tokens.

Fig 3: Merged output-attribution evidence by model for previous outputs of at least 2k tokens. Left: prior output versus

Fig 4: Per-step output-token distribution by provider.

Fig 5: Two ways a prior step’s output can be accounted in

Fig 6: plots these timing metrics against the total input

Fig 2: Per-step append length by step count and by total

Fig 7: Tool call count distribution for Claude and Codex.

Fig 8: Tool-call latency bins by call share and by total-

Limitations

• Dataset limited to two providers and 43 developers; may not capture full diversity of coding agent users or tasks.
• No adversarial stress testing or synthetic workload variation to probe system robustness.
• Raw user messages and tool input/output text are removed for privacy, limiting semantic analyses.
• Analysis focuses on aggregate workload characterization; does not include end-to-end serving system benchmarking or optimizations.
• Only approximate token cost estimation using published API prices; does not account for all infrastructure or engineering costs.
• Output attribution (output cached vs resend) only proxied from token accounting heuristics; internal caching implementations remain largely opaque.

Open questions / follow-ons

• How can insights on tool-call frequency and latency tail distributions inform adaptive scheduling or fused tool invocation interfaces to reduce overhead?
• What are the best KV-cache eviction or compression strategies to handle human-paced idle gaps and long prefix contexts without excessive cost?
• Can append-length-aware prefill routing and semantic tool-latency prediction be operationalized in serving engines to improve efficiency?
• How do different output token reuse strategies impact serving scalability and how can they be optimized across diverse model architectures?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, this paper offers concrete, data-driven insights into how agentic coding assistants interact with LLM services over long sessions involving complex tool usage patterns. Although the context is coding agents, similar workload characteristics—long contexts with short incremental updates, heavy multi-step calls, and skewed tool usage distributions—can arise in multi-turn bot interactions and advanced CAPTCHA challenges. Understanding these workload dynamics, such as high prefix cache read costs and bursty multi-call request structures, can inform design of bot interaction rate limits, challenge difficulty tuning, and resource allocation.

The cross-provider trace data and findings highlight that multi-step autonomous bot behaviors may cause high costs and complex state management due to context growth and tool-calling diversity. Defenses could exploit behavioral fingerprints like long interaction loops or tool latency profiles to detect advanced bots or optimize challenge timing. Additionally, the dataset's approach to anonymization and privacy preservation while retaining token and timing detail offers a model for safe logging in sensitive bot-defense infrastructure. Overall, the work provides a valuable empirical foundation for understanding real agentic bot workloads that go beyond simple CAPTCHA patterns toward more realistic and complex automated interactions.

Cite

@article{arxiv2606_30560,
  title={ TraceLab: Characterizing Coding Agent Workloads for LLM Serving },
  author={ Kan Zhu and Mathew Jacob and Chenxi Ma and Yi Pan and Stephanie Wang and Arvind Krishnamurthy and Baris Kasikci },
  journal={arXiv preprint arXiv:2606.30560},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30560}
}

Read the full paper

• arXiv:2606.30560 (abstract)
• arXiv:2606.30560 (PDF)