ORCA-bench: How Ready Are Language Model Agents for Oncall?

Source: arXiv:2607.28545 · Published 2026-07-30 · By Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal et al.

TL;DR

ORCA-bench is a new benchmark designed to rigorously evaluate the capability of large language model (LLM) agents to perform root cause analysis (RCA) within a realistic oncall setting. Unlike prior benchmarks, ORCA-bench exposes LLM agents to a live OpenTelemetry-instrumented microservice system, providing access to six days of telemetry data (metrics, logs, traces) through real-world APIs (Prometheus, Jaeger, OpenSearch via Grafana) in addition to full source code access. The benchmark contains 1,079 RCA tasks with systematically varied ambiguity in user-facing reports, time-to-detection after incident start, and fault scenario complexity. Expert SREs curated ground-truth symptoms and root causes, and an LLM-based judge was validated by human re-scoring with high reliability (Cohen's kappa_w=0.90). Five state-of-the-art coding agents achieve only up to 25.3% RCA accuracy on medium difficulty tasks and as low as 10.0% on hard tasks, with hallucination rates of 7-40%. Ablations show source-code access is critical, and removing it uniformly degrades performance. Given the 50 GB six-day testbed is far smaller and simpler than production systems, the low accuracy highlights that current LLM agents remain far from ready for reliable oncall RCA in real environments.

Key findings

• Best RCA accuracy of frontier coding agents is 25.3% on medium difficulty and 10.0% on hard tasks across 1,079 tasks using ORCA-bench.
• Hallucination rates range from 7% (DeepSeek-V4-Pro) to 40% (GLM-5) of incident reports naming implausible root causes.
• Removing source code access drops RCA accuracy by 9–16 percentage points and increases hallucination rates substantially for all models.
• Telemetry call error or empty response rates range from 26% to 40% across agents, indicating noisy and incomplete evidence retrieval.
• Agents tend to focus on louder symptoms, often missing simultaneous root causes, reducing exhaustive failure attribution.
• The reported results are on a 50 GB six-day testbed with public, stable code and instrumentation; real production systems are larger and more dynamic, implying the reported accuracies are an optimistic lower bound.
• Claude Fable 5 improves RCA depth and accuracy over Claude Opus 4.7 primarily on medium tasks, suggesting gains are harder to achieve on easy/hard levels.
• Telemetry is primarily accessed via Grafana APIs with models spending 70%+ of commands querying telemetry, but source code usage is less frequent (16-20% of commands), though essential.

Methodology — deep read

1. Threat Model and Assumptions: The adversary is not the focus here; instead, the goal is to evaluate if general-purpose LLM coding agents can perform oncall RCA under realistic conditions. Agents have access to source code and live telemetry APIs but face ambiguous user reports, noisy telemetry data, and time delays since incident onset. The challenge is to correctly identify plausible root causes with limited and noisy inputs. No assumption is made that agents have perfect or privileged knowledge.

2. Data: ORCA-bench is constructed atop the OpenTelemetry Astronomy Shop microservice system, a distributed e-commerce demo with 19 services written in multiple languages. The benchmark provides 6 days of telemetry data totaling 50 GB, including metrics (Prometheus), logs (OpenSearch), and traces (Jaeger), all accessible over realistic Grafana APIs. The dataset contains 1,079 RCA tasks generated by toggling 11 feature flags across 13 scenarios designed and validated with expert SREs. Each task defines: a user-facing report of varying specificity (easy/medium/hard), a report time offset, and a detection time (TTD) sampled from {15 min, 1h, 8h, 24h}. Ground-truth plausible root causes and detailed symptom rubrics per telemetry modality were curated and hand-validated by SREs. A verified subset of 40 tasks has full human annotation.

3. Architecture / Algorithm: The evaluated agents (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, GLM-5, DeepSeek-V4-Pro) operate using the Terminus-2 agent harness, granting them interactive tmux terminal access with telemetry and source code navigable via CLI and Grafana API queries. Agents generate RCA reports based on interactive investigation of telemetry data and source code. A novel evaluation involves an LLM-based judge (GPT-5.4) scoring agent reports against the multi-label ground-truth plausible root causes with a 0-3 rubric reflecting symptom confirmation and root cause identification.

4. Training Regime: Existing pretrained frontier agents were tested without additional training or fine-tuning on ORCA-bench. The manuscript does not mention additional specialized training, adapting weights, or hyperparameter tuning relevant to ORCA-bench.

5. Evaluation Protocol: The evaluation pipeline includes (E1) detection check (did the agent detect presence/absence of incident?), (E2) per-rubric grading (0–3 scale per plausible root cause reflecting investigation depth), and (E3) aggregation into overall RCA Accuracy (exact match of all plausible root causes named), RCA Depth (mean rubric score), and hallucination rate (naming root causes outside plausible set). Scores from the LLM judge were validated against human scorers on the verified subset (Cohen’s kappa_w=0.90). Results involve multiple difficulty and TTD settings and include controlled ablations such as removing source code access.

6. Reproducibility: The public dataset and tasks are released at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench. The live system environment and telemetry interfaces are based on the open source OpenTelemetry demo project. The LLM-as-judge method and agent harness are described but it is unclear if all agent code, their weights, or full evaluation scripts are released.

Example End-to-End: For an incident task with report "Shoppers cannot use their cart," the agent has access to telemetry data up to detection time and can query logs, traces, metrics, and source code terminals. The agent synthesizes these signals to identify root cause hypotheses, cross-checks symptoms, and produces an RCA report citing feature flag faults (e.g., productCatalogFailure). This report is then scored by GPT-5.4 against the curated ground truth rubric, yielding an accuracy and partial credit score reflecting diagnostic success.

Technical innovations

• ORCA-bench provides the first SRE benchmark exposing LLM agents to a full live telemetry stack (metrics, logs, traces) via real-world interfaces plus full source-code access in a coherent oncall RCA environment.
• Systematic task generation jointly varying user report specificity, time-to-detection (TTD), and multiple co-occurring fault scenarios reflects production incident complexity unlike prior single-fault or synthetic fault injection benchmarks.
• Use of a GPT-5.4-based LLM-as-judge with a detailed multi-level rubric for scoring RCA reports, validated against human re-scorers with strong inter-rater reliability (Cohen’s kappa_w=0.90).
• Development of a semi-automated ground-truth symptom curation workflow involving multi-modal telemetry (metrics/logs/traces), frontend symptom replay, and expert SRE validation to produce reliable multi-label RCA ground truth.

Datasets

• ORCA-bench — 1,079 RCA tasks over 6 days, 50 GB telemetry data — Public at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench
• OpenTelemetry Astronomy Shop microservice demo — 19 microservices, 50 GB telemetry — Open source

Baselines vs proposed

• Claude Sonnet 4.6: RCA accuracy (Medium) = 25.3% vs DeepSeek-V4-Pro = 16.8%
• Claude Sonnet 4.6: RCA accuracy (Hard) = 10.0% vs GPT-5.5 = 8.6%
• Removing source code access: RCA accuracy drops by 9–16 percentage points for all models (e.g., Opus 4.7 from ~25% to ~10%)
• Hallucination rate: GLM-5 = 40.2% vs DeepSeek-V4-Pro = 7.2%
• On ORCA-bench Verified, Claude Fable 5 achieves RCA accuracy of 40.6% (±8.8%) compared to GPT-5.5 21.9% (±7.4%)
• RCA depth: GPT-5.5 achieves 48.8% (partial progress score) over 884 incident tasks; other agents lower.

Limitations

• Evaluations are limited to a single 50 GB, 6-day microservice system whose code and telemetry instrumentation are public and static, whereas real systems are larger, dynamic, and idiosyncratic.
• Agents tested are general-purpose pretrained LLM coding agents without specialized training or fine-tuning on ORCA-bench tasks.
• The LLM judge, though validated on a 40-task verified subset, may have limitations in subtle semantic scoring, especially for overlapping root causes.
• No explicit adversarial robustness tests or evaluation under telemetry corruption or evasion attacks were performed.
• The benchmark isolates tasks rather than mimicking continuous multi-incident oncall shifts, potentially missing context switching challenges.
• Some scoring metrics rely on quantity-limited task subsets and manual curation, which may not fully represent all failure modes and RCA complexities.

Open questions / follow-ons

• How can LLM agents be adapted or fine-tuned specifically for multi-modal telemetry reasoning and timed incident investigation to improve RCA accuracy?
• What new architectures or retrieval methods enable better integration of noisy telemetry data with source code to reduce hallucination and improve root cause identification?
• How does performance degrade or adapt in larger-scale, dynamic production environments with evolving code and telemetry instruments?
• Can the LLM-as-judge evaluation framework be extended for multi-incident continuous oncall settings or integrated with human-in-the-loop feedback?

Why it matters for bot defense

ORCA-bench illustrates the substantial gap between current general-purpose LLM coding agents and the real-world requirements for production reliability tasks such as root cause analysis. For bot-defense and CAPTCHA practitioners leveraging language models or AI agents for security or reliability diagnosis, these results caution that LLM hallucination and limited multi-modal reasoning can lead to incorrect or incomplete incident explanations. The benchmark's use of real telemetry interfaces and time-delay factors emphasize that any system relying on LLM agents for dynamic, context-sensitive investigations must rigorously validate against noisy, incomplete, and ambiguous inputs. Providing source code access is essential, but insufficient alone, and consistent grounding in multi-source evidence is critical to reduce false positives/negatives that might undermine security monitoring or attack diagnostics. Practitioners should consider ORCA-bench as a template for realistic multi-modal evaluation and for quantifying readiness before deploying LLM agents in sensitive, high-stakes operational settings.

Cite

@article{arxiv2607_28545,
  title={ ORCA-bench: How Ready Are Language Model Agents for Oncall? },
  author={ Albert Gong and Kyuseong Choi and Abhineet Agarwal and Jason Schechner and Ryan Huang and Raj Agrawal and Anish Agarwal and Raaz Dwivedi },
  journal={arXiv preprint arXiv:2607.28545},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28545}
}

Read the full paper

• arXiv:2607.28545 (abstract)
• arXiv:2607.28545 (PDF)