Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs

Source: arXiv:2607.28573 · Published 2026-07-30 · By Woongkyu Lee, Jungwook Choi

TL;DR

This paper addresses the challenge of improving autonomous computer-use agents (CUAs) deployed locally under strict hardware constraints, focusing on how inference-time scaling affects their performance, efficiency, and failure modes. While prior work shows that inference-time scaling can boost large proprietary CUAs, the authors systematically study its effects on smaller, resource-constrained local models. They evaluate contextual (history length), temporal (max steps), structural (single vs two-stage agents), and parallel (multiple plans) scaling dimensions across several local models (Qwen3-VL-8B/30B, UI-TARS-1.5-7B, OpenCUA-7B) on the OSWorld benchmark comprising 361 Ubuntu GUI tasks.

The key findings reveal that moderate contextual scaling (history length) improves trajectory stability and task accuracy, but returns saturate as token costs rise and failures shift from stalled or repetitive loops to premature false successes where the agent wrongly concludes the task is done. Temporal scaling reduces max-step stalls but fails to improve actual task success, often just extending erroneous trajectories. Structural decomposition into two-stage planning and grounding agents introduces significant overhead and format-parsing failures that reduce accuracy relative to single agents. Parallel scaling of multiple plans partially alleviates these issues but at a sub-linear accuracy gain relative to sharply higher computational cost.

Overall, the work argues that naive, uniform inference-time scaling is inefficient for local CUAs. Instead, efficient local agents require selective compute allocation, failure-aware control mechanisms, and agentic frameworks designed around the reasoning limits and operational costs of local open-source models. This systematic and detailed empirical study fills a gap by clarifying how various inference-time scaling methods perform and fail in resource-constrained local deployment scenarios.

Key findings

• Providing minimal history context (H=1) boosts single-agent task success from ~18% to over 25% on OSWorld; gains continue up to H=4, then saturate.
• Increasing history length reduces repetitive action loops and max-step stalls but shifts failures toward premature false successes, capping peak accuracy at 28.56% for H=4.
• Extending maximum decoding steps (S) from 15 to 100 yields minimal increases in task success, but increases operational cost linearly and shifts failure modes toward false completions.
• Two-stage agents (planning + grounding) achieve lower success rates than equivalent single-agent baselines while incurring substantially higher token costs due to planning overhead and format-parsing failures (non-trivial in failure cohorts).
• Increasing the number of parallel candidate plans (P) in two-stage agents partially recovers accuracy and reduces format errors but introduces sharply rising token usage and sub-linear gains relative to computation.
• Contextual scaling most effectively stabilizes trajectories, while temporal scaling tends to extend erroneous paths rather than correct them in local models.
• Structural decomposition designed for larger models can introduce additional failures and overhead in local CUAs due to limited reasoning and format compliance of small open-source models.
• Selective allocation of compute resources, rather than blind scaling, is necessary to optimize local CUAs under resource constraints.

Threat model

Indirect threat model focusing on local agent reasoning and computational constraints rather than active adversaries. Assumes an agent operating autonomously on local hardware with limited inference-time budgets, challenged solely by its own reasoning limits, input complexity, and token cost constraints. No external attackers or data poisoning considered.

Methodology — deep read

The paper analyzes inference-time scaling effects on local computer-use agents (CUAs) through systematic empirical evaluation focused on four scaling dimensions: contextual (history length, H), temporal (max decoding steps, S), structural (single agent vs two-stage planning + grounding), and parallel (number of candidate plans, P).

Threat model and assumptions: The study assumes local CUAs running on constrained hardware (e.g., a single A100 GPU) without auxiliary privileged interfaces like accessibility trees. The adversary model is implicit—focusing on the agent's own reasoning and computational limits rather than external attacks. The local models are open-source variants, smaller than cutting-edge proprietary agents, emphasizing operational constraints.

Data and evaluation protocol: The benchmark is OSWorld, a realistic GUI environment consisting of 361 real-world Ubuntu tasks (desktop interactions from screenshots only). Evaluation excludes tasks requiring unsupported services. Metrics include task success rate, step counts per task, and cumulative prompt token usage per task to capture computational cost. Failure modes (e.g., max-step stalls, repetitive loops, premature false successes, format errors) are analyzed.

Models evaluated include four single-agent MLLMs: Qwen3-VL-8B-Instruct, Qwen3-VL-30B-A3B-Instruct (MoE), UI-TARS-1.5-7B, OpenCUA-7B, and a two-stage agent combining Qwen3 planners (8B and 30B-A3B) with GTA-1-7B grounding.

Architectures:

1. Single agent: A multimodal LLM takes current screenshot, instruction, and history (H screenshots/actions) to predict next actions up to max steps S.
2. Two-stage agent: Decouples planning (generates multiple candidate plans P) and grounding (executes selected plan) stages. Planning outputs code-like plans which grounding translates into GUI actions.

Scaling variables:

• Contextual scaling (H): History length varied {0,1,4,8} representing number of prior screenshots and actions given as context.
• Temporal scaling (S): Max decoding steps tested at 15, 50, 100 step limits to allow longer episode horizons.
• Parallel scaling (P): Number of candidate plans in two-stage agent varied from 1 to 4.

Training regime is not detailed (models presumed pre-trained). All inference conducted on Nvidia A100-80GB GPU using vLLM for token accounting.

Evaluation protocol assesses performance-cost tradeoffs, failure mode cohort shifts, and compares single vs two-stage architectures across scaling dimensions. Qualitative trajectory analysis illustrates behavior changes.

Reproducibility is partially supported; evaluated models are referenced from prior public works with some open-source availability (e.g., Qwen3-VL and UI-TARS). OSWorld benchmark is publicly described, though full code/data release status is unclear.

Technical innovations

• Systematic categorization of inference-time scaling into contextual (history), temporal (max steps), structural (agent decomposition), and parallel (multi-plan) dimensions for local CUAs.
• Comprehensive failure-mode analysis showing inference-time scaling alters failure distribution (from repetitive/stalls to false successes) rather than uniformly improving accuracy.
• Empirical demonstration that structural two-stage decomposition introduces format-parsing and planning overhead failing to improve performance in local open-source CUAs compared to simple single-agent setups.
• Quantitative cost-accuracy tradeoff characterization revealing sub-linear gains of parallel plan generation relative to computational expense, guiding selective compute allocation.

Datasets

• OSWorld benchmark — 361 real-world Ubuntu GUI tasks — public benchmark (Xie et al., 2024)

Baselines vs proposed

• Qwen3-VL-30B-A3B single agent (H=4, S=15): task success ≈ 28.56% vs H=0 baseline ≈ 18%
• Increasing max steps S from 15 to 100 on Qwen3 models: task success gain < 5%, while token cost and steps per task increase linearly
• Two-stage agent with Qwen3-30B planner + GTA-1 grounding (P=1, H=4, S=15): task success lower by ~5-10% vs single agent baseline at similar compute cost
• Two-stage agent increasing candidate plans P from 1 to 4: task success improves partially (~3-5%) but token usage grows ~3-4x, showing sub-linear returns
• UI-TARS-1.5-7B single agent shows similar qualitative trends in failure mode shifts across contextual and temporal scaling

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28573.

Fig 1: Illustration of CUA frameworks and inference-time scaling dimensions. (a) The

Fig 2: Impact of history length on single-agent performance, efficiency, and failure modes.

Fig 3: Comparison of agent trajectories by contextual scaling. Without contextual

Fig 5: Impact of parallel scaling on two-stage agent. (a,b) Task success rates of Qwen3-8B

Limitations

• Evaluation limited to Ubuntu GUI tasks in OSWorld; generalization to other OSes or GUIs unclear.
• Only open-source, moderately sized models assessed; results may not fully apply to much smaller or larger local models.
• No adversarial attack modeling; failure modes analyzed are intrinsic to model reasoning capacity rather than malicious interference.
• No training or fine-tuning experiments included to improve two-stage agent coordination or format compliance.
• Temporal scaling tested only with fixed step limits; dynamic step control or early termination strategies not explored.
• Reproducibility limited by absent explicit code release or pretrained checkpoints for some models.

Open questions / follow-ons

• How can local agents implement failure-aware control mechanisms to detect and recover from premature false successes or stalled progress?
• What memory management or context compression techniques can provide sufficient historical grounding with minimal token cost in local CUAs?
• Can specialized fine-tuning or architectural modifications reduce planning and format-parsing failures in two-stage local agents?
• How do adaptive or dynamic inference-time scaling policies based on intermediate feedback impact efficiency and accuracy tradeoffs?

Why it matters for bot defense

This work provides bot-defense and CAPTCHA practitioners with detailed insights into the practical limitations and failure patterns of local computer-use agents scaling inference-time resources. The observation that additional compute often changes failure modes without linear performance gains warns that attackers relying on more computation at runtime may still face inherent reasoning bottlenecks and unique failure symptoms.

For CAPTCHA design, understanding how resource-constrained agents trade off effort, stability, and premature task termination can guide challenge designs that exploit these fail modes or impose selective compute burdens. The revealed inefficiencies of structural decomposition and parallel scaling also indicate that simple agentic defenses might suffice, as more complex parallel attempts are costly and yield diminishing returns on local models. Overall, the paper suggests that to robustly guard against locally deployed autonomous agents, bot-defense strategies should account for non-linear scaling effects, enforce bounded compute budgets, and develop adaptive tests that trigger failure-aware mitigation.

Cite

@article{arxiv2607_28573,
  title={ Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs },
  author={ Woongkyu Lee and Jungwook Choi },
  journal={arXiv preprint arXiv:2607.28573},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28573}
}

Read the full paper

• arXiv:2607.28573 (abstract)
• arXiv:2607.28573 (PDF)