TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

Source: arXiv:2608.06346 · Published 2026-08-06 · By Yunjia Qi, Zehua Yin, Xintong Shi, Hao Peng, Songyuanyi Lu, Yixian Liu et al.

TL;DR

The paper addresses the challenge of identifying the earliest critical error responsible for a task failure in long-horizon trajectories generated by LLM-based agents. Existing approaches struggle due to distributed evidence across many prior steps and the coexistence of multiple local errors with differing downstream impacts—some repaired, some irrelevant, and only some causing the final failure. The authors propose TRAJDEBUG, a multi-stage framework that tackles these challenges by compressing lengthy trajectory context into multi-granularity views, detecting evidence-grounded error triggers, grouping these into error instances with lifecycle state classification, and performing causal attribution to select the terminal-impacting critical error. They further construct TRAJERRBENCH, a new large benchmark of 486 manually annotated failed trajectories spanning tool-use and long code-debugging scenarios with steps averaging from 29 to 120. Extensive experiments on multiple benchmarks show TRAJDEBUG outperforms prior prompting and multi-agent diagnostic baselines by a substantial margin, especially on long trajectories. Application studies demonstrate that TRAJDEBUG’s diagnoses enable actionable feedback that improves downstream agent success by up to 10.8% in per-run repairs and 5.7% in zero-shot failure-memory transfer. The code and dataset will be released to support future research.

Key findings

• TRAJDEBUG achieves a macro-average critical error step detection accuracy of 34.11% across 7 diverse datasets, outperforming direct prompting with the same backbone by 8.42 percentage points (from 25.69% to 34.11%).
• On long-horizon benchmarks like ALFWorld (60 steps) and SWE-Bench Pro (120 steps), TRAJDEBUG improves accuracy by +13.0% and +6.97% respectively over strong direct prompting baselines.
• The new TRAJERRBENCH benchmark consists of 486 failed trajectories with an average step count ranging from 29.27 (tau2-Bench) to 119.7 (SWE-Bench Pro), significantly increasing trajectory length and complexity for critical error detection evaluation.
• Ablation shows multi-granularity compression is critical, with accuracy dropping by 21.02 points when replaced by standard truncation. Removing error trigger detection or error state classification reduces accuracy by 4.99 and 7.31 points respectively, confirming all components contribute substantively.
• Local error density grows with trajectory length, averaging 7.62 local errors per failed trajectory, but only one critical error exists per trajectory. Among non-critical errors, 61.9% are repaired later, 31.4% persist harmlessly, and 6.6% are dormant, motivating error lifecycle classification.
• TRAJDEBUG maintains more stable accuracy across trajectory length buckets than other methods. It retains over 20% detection accuracy on the longest trajectories (141+ steps) where the best baseline degrades below 14%.
• When TRAJDEBUG’s failure diagnoses are injected as feedback into a downstream GLM-5.1 agent to retry failed tasks, success rates improve by 10.8% on average; failure-memory transfer experiments yield a 5.7% average success gain on held-out tasks.

Threat model

The threat model assumes an adversarial setting where the agent executes a complex task resulting in failure due to cascading or latent errors across a long sequence of actions. The system only observes the failed trajectory and final failure signal without access to successful traces, oracle corrections, or external debugging information. The adversary is not explicitly a human attacker but represents failures intrinsic to the agent's reasoning or environment unpredictability. The method cannot intervene mid-trajectory or re-run parts of the environment; it must diagnose post hoc using only the recorded trajectory.

Methodology — deep read

TRAJDEBUG addresses critical error detection in failed agent trajectories by decomposing the problem into stages that leverage explicit evidence and lifecycle reasoning.

1. Threat model & assumptions: The adversary is the environment or agent failure causing a task to be unsuccessful after a long trajectory of reasoning, planning, action, and observation steps. The agent receives only the failed trajectory and no success traces or gold signals. The method must identify the earliest 'decisive error' step causally responsible for the final failure despite multiple coexisting local errors, some repaired or harmless.

2. Data: The authors constructed TRAJERRBENCH with 486 manually annotated failed trajectories from τ2-Bench (400 trajectories, avg length 29.3 steps) covering diverse tool-use scenarios, and SWE-Bench Pro (86 trajectories, avg length 119.7 steps) for long-horizon coding/debugging tasks. Annotation protocol involved 3 annotators per trajectory with high inter-annotator agreement (Fleiss' κ=0.91 for τ2-Bench, 0.67 for SWE-Bench Pro). Failure step labels separate decisive errors from repaired or dormant mistakes.

3. Architecture / algorithm: TRAJDEBUG consists of four stages:

• Multi-granularity compression: For each trajectory step, constructs three views (high-detail with original instructions and reasoning, medium-detail summaries, and low-detail progress snapshots) to preserve key evidence across long contexts while enabling efficient grounding.
• Error Trigger Detection: Uses an LLM-based detector to identify per-step atomic error triggers—local evidence-grounded mismatches denoted by the erroneous commitment, the violated reference object, reference category (task conflict, history conflict, intra-step conflict, environment anomaly), and execution phase (planning, reasoning, action, observation, verification). Each trigger requires verbatim evidence citations from trajectory or instructions.
• Error State Classification: Groups triggers violating the same reference object into error instances representing persistent wrong commitments. Classifies each instance's lifecycle state by two judgments: whether the error is resolved or remains active, and whether it leaves a terminal footprint (irreversible state change, persistent violation, or costly recovery after >50% of trajectory). Four states arise: Clean Resolution, Costly Resolution, Manifest Active, Latent Active. Only terminal relevant instances (Costly Resolution, Manifest Active) are passed for attribution.
• Candidate-Set-Guided Causal Attribution: An LLM-based attribution head ranks candidate error instances by integrating their first step, state label, and supporting evidence to select the critical error step causally explaining the final failure. The earliest error heuristic is insufficient due to budget debt effects.

4. Training regime: The paper does not specify additional training; the framework primarily uses prompting and organization of evidence with a fixed LLM backbone (Qwen3-235B-A22B-Thinking) for fair comparisons.

5. Evaluation protocol: Evaluated on multiple benchmarks (WhoAndWhen, AgentDebugBench, TRAJERRBENCH) totaling 869 trajectories, measuring exact critical error detection step accuracy. Methods receive only failed trajectories without success traces or auxiliary signals. Ablations test impact of component removal on τ2-Bench and SWE-Bench Pro. Length bucket analysis measures scaling with trajectory length. Application studies use the diagnoses as feedback in re-execution and failure-memory transfer scenarios, measuring task success rate improvements.

6. Reproducibility: The authors will release code and datasets. The paper uses publicly known benchmarks and newly constructed TRAJERRBENCH with detailed annotation protocols.

Example end-to-end: Given a failed trajectory ~100 steps (e.g. from SWE-Bench Pro), TRAJDEBUG first compresses the trajectory into multi-level views, then applies error trigger detection to find explicit mismatches with task instructions or history context at individual steps. It groups repeated triggers violating the same reference into error instances and classifies each according to resolution and terminal footprint. Finally, it feeds candidate error instances into an attribution LLM prompt that selects the earliest decisive error step causing the final failure. This step’s evidence and classification enable trustworthy localization even with multiple local errors and repairs.

Technical innovations

• Multi-granularity trajectory compression preserving local evidence while compacting distant context to enable scalable grounding in long-horizon trajectories.
• Evidence-grounded error trigger detection that requires explicit, citable evidence linking local agent commitments to violated references, reducing hallucinated or weakly formed diagnoses.
• Error lifecycle classification of error instances into resolved vs active and terminal footprint bearing categories, enabling disambiguation among multiple coexisting local errors.
• Candidate-set-guided causal attribution using an LLM to select the critical error step based on error states and supporting verbatim evidence rather than simple heuristics or holistic judgment.

Datasets

• TRAJERRBENCH — 486 failed trajectories covering realistic tool-use and long-horizon coding tasks — constructed from 400 τ2-Bench trajectories and 86 SWE-Bench Pro trajectories

Baselines vs proposed

• Direct Prompting (Qwen3-235B-A22B-Thinking backbone): Accuracy = 25.69% vs TRAJDEBUG: 34.11%
• Direct Prompting Gemini-3.1-Pro on ALFWorld: 42.06% vs TRAJDEBUG: 48.41%
• AgentDebugger on AgentDebugBench: 23.72% vs TRAJDEBUG: 34.11%
• CHIEF on SWE-Bench Pro: 2.32% vs TRAJDEBUG: 24.41%
• AgentRX on ALFWorld: 8.00% vs TRAJDEBUG: 48.41%
• Ablation w/o Multi-Granularity Compression: 17.56% vs full TRAJDEBUG: 38.58%
• Ablation w/o Error Trigger Detection: 33.59% vs full TRAJDEBUG: 38.58%
• Ablation w/o Error State Classification: 31.27% vs full TRAJDEBUG: 38.58%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06346.

Fig 1: Critical error detection requires grounding er-

Fig 3: An overview of the framework of TRAJDEBUG.

Fig 4: Relative position of the critical error step

Fig 2: Critical error detection accuracy and local

Fig 5 (page 4).

Fig 5: shows critical-step detection accuracy

Limitations

• Absolute detection accuracies remain low (below 40%), indicating critical-step localization from only failed trajectories remains very challenging.
• The approach relies on large instruction-tuned LLMs and explicit verbatim evidence, which may miss implicit or latent failures without clear textual contradictions.
• Error state classification thresholds (e.g., 50% budget debt cutoff) and categories are heuristically set, which might limit generalization to other domains or trajectory types.
• While the benchmark is significantly larger and longer than prior datasets, it still covers a limited set of realistic tool-use and code-debugging scenarios and may miss other complex agent environments.
• The evaluation assumes access to reliable human annotation of critical error steps, which may not scale or be available in fully automated deployment.
• The method requires multiple passes over different granularities of historical context, which may be computationally intensive for extremely long trajectories.

Open questions / follow-ons

• How to improve interpretability and traceability of implicitly latent errors that do not produce direct verbatim evidence?
• Can this framework be extended to real-time or online critical error detection with partial trajectories?
• How sensitive are the error state classifications and causal attribution decisions to the choice of LLM or prompt design?
• How well does the method generalize to non-textual agent modalities or uncertain/noisy observations beyond current benchmarks?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, TRAJDEBUG offers a comprehensive framework to analyze failed agent trajectories in complex multi-step tasks by explicitly grounding error identification in verifiable evidence distributed over long contexts. This approach can inform diagnostics and debugging of bot-like behaviors that fail due to cascading decision errors rather than single-step mistakes. The error lifecycle classification helps distinguish transient or repaired errors from terminal failures, enabling more reliable fault localization useful for developing robust mitigation strategies. Additionally, the demonstrated utility of diagnostics as feedback to improve agent success suggests potential for adaptive defenses that learn from failure patterns. However, the framework's reliance on large LLMs and extensive trajectory logs may limit feasibility for real-time CAPTCHA or anti-bot integrations without further optimization.

Cite

@article{arxiv2608_06346,
  title={ TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories },
  author={ Yunjia Qi and Zehua Yin and Xintong Shi and Hao Peng and Songyuanyi Lu and Yixian Liu and Richeng Xuan and Yuhong Liu and Zhichao Hu and Xiaozhi Wang and Lei Hou and Bin Xu and Juanzi Li },
  journal={arXiv preprint arXiv:2608.06346},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.06346}
}

Read the full paper

• arXiv:2608.06346 (abstract)
• arXiv:2608.06346 (PDF)