Falsifiable Commitment Planning for Self-Correcting Web Agents

Source: arXiv:2607.24167 · Published 2026-07-27 · By Guangyi Liu, Huan Zhao, Quanming Yao

TL;DR

FCPAgent addresses the challenge of long-horizon web agents going off track during multi-step web navigation tasks before ultimately failing. Traditional agents either plan or reflect but lack explicit runtime checks on when to trust or distrust intermediate plan steps, making them vulnerable to silent drift and compounding errors. This paper introduces falsifiable commitment planning, where each plan step is not just an instruction, but a testable commitment with explicit confirming and falsifying evidence and an associated confidence. FCPAgent executes plans via a plan-test-repair loop that detects deviations online through hybrid evidence matching and LLM-based verification. When a commitment is falsified, a scope-aware repair revises the smallest failing component (execution, skill, or plan level). Experiments on WebArena show a 13.8% relative success rate improvement over the strongest baseline, particularly on long-horizon tasks. Furthermore, zero-shot evaluation on WebChoreArena demonstrates generalization gains, and ablations confirm the importance of falsifiable commitments, hybrid testing, and repair in improving robustness and efficiency.

Key findings

• FCPAgent achieves 65.3% success on WebArena, a 13.8% relative improvement over ColorBrowserAgent's 57.4%.
• Success gains are largest on GitLab tasks, improving from 63.4% to 75.5% (+19.1%).
• Zero-shot transfer to WebChoreArena yields 31.7% success, a 9.6% relative improvement over ColorBrowserAgent's 28.9%.
• Long-horizon tasks show disproportionate gains: for Shopping with 11+ steps, improvement reaches 161% relative over baseline; for GitLab long tasks, gains are +87%.
• Ablation removing falsifiable planning drops average success from 70.6% to 64.4%, the largest impact among components tested.
• Hybrid commitment testing reduces average LLM call time by 19.5% compared to slow-only testing without affecting success.
• Experience scaling shows that incorporating failure-repair memories improves success over using skill library alone.
• The multi-scope repair mechanism localizes problems at execution, skill, or planning level enabling targeted recovery rather than full replanning.

Threat model

The adversary consists of the complex, partially observable, and dynamic web environment, which can subtly cause the agent's plan execution to drift away from the intended subgoals via plausible but invalid states or actions. The environment may change or present states that violate assumptions underlying skills or plans, leading to silent degradation without immediate failure signals. The agent cannot directly observe the true latent state or receive immediate success feedback for intermediate steps, so it must detect deviations from the prescribed commitments through observable evidence. The adversary does not actively manipulate inputs maliciously but exploits environment uncertainty and complexity.

Methodology — deep read

1. Threat Model & Assumptions: The adversary modeled is the partially observable, dynamic web environment and task complexity that can cause the agent to drift off the valid execution path silently. The adversary is not an active adversarial attacker but the inherent environment complexity, so the agent must identify when progress assumptions become invalid. The agent cannot forcibly observe true latent states but must rely on observable browser content such as accessibility trees, screenshots, and web page labels. 2. Data: Evaluation uses WebArena, a benchmark of 6 realistic web applications with natural language instructions and success validators, plus WebChoreArena for zero-shot testing. Training trajectories produce successful skills and failure-repair memories forming reusable libraries. Dataset sizes and splits are not fully specified but follow prior work. 3. Architecture / Algorithm: FCPAgent represents each plan step as a Falsifiable Commitment Unit (FCU) consisting of (subgoal, linked reusable skill, confirming evidence, falsifying evidence, confidence score). Plans become sequences of FCUs. Confirming evidence includes pre-conditions, progress evidence, and completion signals. Falsifying evidence is hierarchically organized into execution-level, skill-level, and planning-level falsifiers. Execution loops follow a plan-test-repair process: Before executing an action, a lightweight test checks if the action likely sustains the FCU. After action execution, browser states are checked against confirming/falsifying evidence using a hybrid tester combining fast evidence matching (textual entailment via nli-deberta-v3-base and image-text matching via SigLIP2) with an LLM-based diagnostic verifier for uncertain or risky cases. The LLM considers task instruction, active FCU, recent trajectory, current observations, and tester signals to decide whether to continue, advance FCU, or trigger repair. Repair uses scope-aware, experience-enhanced revision: based on falsifier diagnosis, the system localizes the contradiction to execution (minor correction), skill (switching skills), or plan level (replanning or restarting from a safe state). The repairer retrieves analogous prior failures to guide recovery but bases corrections on current state and libraries. 4. Training Regime: The skill library and failure-repair library are distilled offline from training trajectories gathered on WebArena tasks. These libraries remain fixed during evaluation to ensure fair comparison. The backbone LLM is Qwen3.5-397B-A17B, accessed via API with temperature=1. Hyperparameters include top-K=3 skills and P=2 failure experiences retrieved per task, and λ=0.8 balancing text and image matching scores. 5. Evaluation Protocol: The main metric is task success rate on WebArena across six domains (Shopping, Admin, GitLab, Reddit, Map, Cross). Zero-shot evaluation is performed on WebChoreArena without additional training on that benchmark. Ablation studies vary falsifiable planning, hybrid testing, and repair components. Efficiency is measured by LLM call time per task. Long-horizon robustness is analyzed by success stratified by task length buckets defined by the baseline's trajectory length. 6. Reproducibility: Code and detailed dataset splits are not specified as released. The paper uses known public benchmarks (WebArena, WebChoreArena) but the failure-repair library is constructed from training logs that are fixed for evaluation. The core method is implemented on top of Qwen3.5 backbone via API calls.

Technical innovations

• Formulation of falsifiable commitment units (FCU) that attach explicit confirming and falsifying runtime evidence for each plan step in web agents, enabling online validity testing rather than implicit execution steps.
• A hybrid commitment testing mechanism combining lightweight evidence pattern matching with selective LLM-based diagnostic verification to efficiently detect execution drift and false commitments.
• Scope-aware commitment repair that uses diagnosed falsification evidence to localize failure at execution, skill, or planning level, enabling fine-grained targeted recovery from failures rather than wholesale replanning.
• Experience-enhanced planning and repair via retrieval of successful skill procedures and failure-repair memories that encode both positive and negative priors to guide both commitment generation and recovery.

Datasets

• WebArena — multi-domain realistic web task benchmark with 6 web applications — public
• WebChoreArena — extended WebArena with more labor-intensive tasks for zero-shot evaluation — public

Baselines vs proposed

• ColorBrowserAgent: success = 57.4% vs FCPAgent: 65.3% on WebArena average success
• ColorBrowserAgent: 28.9% vs FCPAgent: 31.7% zero-shot success on WebChoreArena average
• Ablation without falsifiable planning: 64.4% vs full FCPAgent 70.6% average over Shopping, Admin, GitLab
• Hybrid commitment testing LLM call time: 374s vs slow-only testing: 464.5s per task on Shopping+Admin

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24167.

Fig 1: An off-track execution example. An ordinary plan

Fig 2: FCPAgent casts web-agent execution as online testing of falsifiable commitments. Hybrid commitment testing checks

Fig 3: Success rate (%) by task length. The strongest gains

Fig 4: GitLab success rate as the amount of training

Fig 5: A WebArena-Shopping case study showing how FCPAgent detects plan-level drift, replans the active commitment, and

Fig 6 (page 7).

Fig 7 (page 7).

Fig 8 (page 7).

Limitations

• The approach depends heavily on the quality and granularity of generating confirming and falsifying evidence; overly broad falsifiers may cause unnecessary repairs, overly narrow ones may miss deviation.
• The current framework is tested only on scripted web environments without adversarial or malicious web states; robustness under adversarially crafted inputs is unexamined.
• LLM-based verification relies on the capability of the underlying LLM (Qwen3.5-397B-A17B) which may limit generalization or interpretability.
• Failure-repair memory usage is limited to experience encountered in training; unobserved failures or novel failure modes may not be reliably handled.
• No end-to-end reproduction package or frozen weights currently provided, limiting exact result replication outside a closed experimental setup.
• The benchmark evaluation uses heuristic task length buckets from baseline trajectories, which may introduce confounding correlations in long-horizon task analysis.

Open questions / follow-ons

• How to automatically learn or generate falsifying and confirming evidence patterns rather than relying on hand-designed or experience-based priors?
• How early and reliably can falsifiable commitments detect deviations before failures compared to existing reflection or planning methods?
• Can falsifiable commitment planning be extended to adversarial or malicious web environments where deceptive states may appear?
• What is the impact of different backbone LLM architectures and sizes on the diagnostic verification performance and overall robustness?

Why it matters for bot defense

From a bot-defense or CAPTCHA practitioner perspective, this paper's falsifiable commitment planning framework offers a novel approach to strengthen the reliability of autonomous web agents operating under uncertain, dynamic conditions. By making each plan step explicitly testable via observable evidence and enabling online detection of execution drifts, systems can reduce silent off-track behaviors that might degrade automation or introduce exploitable inconsistencies. The hybrid testing approach also balances efficiency and accuracy in runtime validation, which is crucial in latency-sensitive web environments. Furthermore, the scope-aware repair mechanism could inspire more granular correction layers in bot workflows rather than costly full restarts.

Practitioners developing CAPTCHA-resistant bots or detection systems might study falsifiable evidence structures and runtime monitoring tactics to understand how agents reason about plan validity and self-correct with minimal performance penalty. This approach emphasizes the importance of monitoring intermediate execution validity, which may inform design principles for both offensive (bot) and defensive (bot-detecting) systems that rely on modular, observable subgoal commitments. However, the approach presumes strong access to browser state and relies on large language models for diagnosis, which may impact deployment feasibility in highly constrained or adversarial settings.

Cite

@article{arxiv2607_24167,
  title={ Falsifiable Commitment Planning for Self-Correcting Web Agents },
  author={ Guangyi Liu and Huan Zhao and Quanming Yao },
  journal={arXiv preprint arXiv:2607.24167},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24167}
}

Read the full paper

• arXiv:2607.24167 (abstract)
• arXiv:2607.24167 (PDF)