The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
Source: arXiv:2607.22520 · Published 2026-07-24 · By Darshan Tank, Baran Nama
TL;DR
This paper investigates why adding procedural skills to large language model (LLM) agents can both help and hurt task performance, a phenomenon termed the "regression tax." Prior work often measures skill addition by average success rate improvement, but this hides the substantial cost of regressions—tasks solved before skills that become failures after skills are introduced. The authors conduct a large-scale paired evaluation comparing agents with and without skills across nearly 6,000 task runs on two realistic office automation benchmarks (OfficeQA-Pro and SpreadsheetBench) using three different model-harness stacks. They decompose the net effect of skills into gains (newly solved tasks) and regressions (tasks newly broken) and find that regressions offset about 59% of gross gains overall, reducing net improvements significantly.
They identify three distinct mechanisms driving regressions: (i) skill-description osmosis, where the mere presence of a skill description in context—without invocation—alters agent behavior negatively; (ii) grounding displacement, where a skill’s procedural guidance overrides correct input interpretation; and (iii) verification displacement, where skills suppress output verification checks that the agent would otherwise perform. These mechanisms explain why some skills break previously solved tasks. Analysis of persistent residual failures also points to insufficient support at the grounding and verification stages, while existing skills disproportionately target the procedural method stage. The authors re-grade many SpreadsheetBench tasks with a stronger execution check and demonstrate that much verification displacement is recoverable. Their work advocates decomposing skill evaluations by gains and regressions and focusing future skill design on better grounding and verification support rather than procedural guidance alone.
Key findings
- Across 5,832 paired runs, 553 gain transitions were observed but 324 regression transitions erased 59% of those improvements.
- On OfficeQA-Pro, regressions canceled 66% of gains; on SpreadsheetBench, regressions canceled 56% of gains.
- Three regression mechanisms explain failures: 72.8% were grounding displacement, 17.3% skill-description osmosis, and 3.7% grounding plus verification.
- Presence-only (non-invoked) skill descriptions affected outcomes (osmosis), altering behavior without skill invocation.
- Verification displacement accounted for many SpreadsheetBench residual failures; 34% of formula failures were actually correct but marked fail by a shallow grader.
- Regrading SpreadsheetBench tasks with full execution checks recovered +11 to +49 task passes depending on stack and library.
- Existing skill libraries emphasize procedural guidance but under-support grounding and verification, the stages driving most regressions and residual failures.
- Only a single stack-benchmark combination showed statistically significant net skill gains after correction (Claude Code · sonnet-4.6 on SpreadsheetBench), underscoring the regression tax’s importance.
Threat model
The study assumes a benign setting with no adversarial interference. The adversary is effectively the internal behavioral shifts caused by adding procedural skills to an LLM agent. Skills are not manipulated maliciously; rather, the focus is on unintended negative side effects (regressions) that arise purely from how skills influence agent input interpretation, output checks, or context presence. The adversary does not have direct external control or knowledge beyond the agent’s context and skills.
Methodology — deep read
The authors study the costs and benefits of adding procedural skills to LLM agents via paired experiments where the same agent and tasks are run with and without skill libraries. Their threat model concerns an unmodified adversary-free setting: the focus is on understanding intrinsic agent behavior changes when skills are added.
Data comes from two office automation benchmarks designed to stress grounding, procedural, and verification capabilities: OfficeQA-Pro, a complex question answering task over U.S. Treasury finance documents involving long PDFs, tables, and multi-step reasoning; and SpreadsheetBench, real-world Excel tasks requiring workbook edits from natural-language instructions. OfficeQA-Pro grading is numeric tolerance based; SpreadsheetBench grading compares target cell values with the golden workbook but includes a known artifact failing correct formulas that cannot be evaluated by the limited grading engine.
They evaluate three model–harness stacks: OpenCode with MiniMax-M2.7, Codex with GPT-5.4-mini, and Claude Code with Sonnet-4.6, each paired with three independently created skill libraries (Anthropic’s benchmarked iterative, OpenAI’s single-pass scaffolded, and a novel self-critique pipeline). Skill libraries vary in size (3 to 23 skills) but all target the same failure signals derived from baseline no-skill runs.
Skill libraries consist of a description (always present in system context) and an instructional procedure invoked as needed. The authors separate presence-only effects (osmosis) from invocation effects via detailed log trajectory analysis. Agent outcomes are binary pass/fail per task, yielding four paired outcome types: gain, regression, residual failure, retained. Statistical significance is measured by McNemar’s exact test on discordant pairs (regressions vs gains) with multiple comparison corrections.
They qualitatively label regressions by reviewing paired trace divergences and divide them into osmosis (no invocation, behavior shifts via description), grounding displacement (procedure overrides correct input parsing), verification displacement (procedure suppresses output checks), or other/unclear. Regrading SpreadsheetBench tasks with a full Excel engine isolates verification failures due to grader artifacts.
The evaluation is comprehensive over 5,832 runs (486 tasks × 4 conditions × 3 stacks) but uses single-run per task without repeated seeds, limiting variance estimates. Each comparison holds model, harness, tasks constant; only the skill library varies. Cross-library contrasts are mined for convergent evidence on regression causes. Code and dataset details are not fully public; evaluation is reproducible given benchmarks and skill sets.
Concretely, the end-to-end method involves recording baseline no-skill trajectories on office tasks, extracting failure signals, authoring skills with three methods, rerunning the same tasks with these skill libraries loaded, logging agent trajectories, grading tasks with standard and enhanced checkers, pairing outcomes by task to decompose net effect into gains and regressions, analyzing trace divergences to classify regression mechanisms, and correcting SpreadsheetBench grades to account for formula evaluation limitations. Example case studies illustrate how skills redirect input grounding or suppress output verification leading to failure, or how presence-only skill descriptions produce behavioral shifts without invocation.
Technical innovations
- A paired evaluation methodology that decomposes skill library effects into gains (new passes) and regressions (new failures) rather than just average success improvements.
- Identification and formalization of three regression mechanisms—skill-description osmosis, grounding displacement, and verification displacement—that explain why skills can break previously solved tasks.
- Application of invocation vs presence-only analysis to isolate how non-invoked skill descriptions influence agent behavior (osmosis).
- Use of full execution-based regrading of SpreadsheetBench tasks to correct for grader artifacts and reveal verification displacement recoveries.
- Cross-library contrasts from three different skill-authoring pipelines applied to the same signals, demonstrating that skill wording and structure impact regression patterns.
Datasets
- OfficeQA-Pro — 94 tasks — curated subset of U.S. Treasury financial question answering benchmark
- SpreadsheetBench — 392 tasks — Excel formula and workbook-editing tasks from real Excel forum problems
Baselines vs proposed
- Without skills baseline: pass rates vary by stack and benchmark (e.g., 51.1% OfficeQA-Pro OpenCode, 70.2% SpreadsheetBench Claude Code)
- Best skill libraries improved pass rates with net positive effects but often high regressions: e.g. Claude Code sonnet-4.6 on SpreadsheetBench gains +43 to +47 tasks, regressions 16 to 20; net +11.0 to +12.0 percentage points (p<.001 after correction)
- Many net effect deltas statistically insignificant after Bonferroni correction, except in Claude Code on SpreadsheetBench
- Execution-based regrading of SpreadsheetBench recovery: e.g. GPT-5.4-mini +49 additional passes recovered by correcting grading artifact
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.22520.
Fig 1: Three stages of an agent task: grounding (reading the right inputs), method (the procedure), and verification (checking
Limitations
- Skill libraries differ in size, wording, and structure, confounding controlled dose effects on results.
- Single-run experiments per task limit estimation of run-to-run variance and robustness.
- Skill classification of regression mechanisms is observational and assigned by a single author; no dual coding or inter-rater reliability reported.
- SpreadsheetBench grading artifacts obscure some failure analysis; correction relies on external engine not built into agent loop.
- Study is limited to office automation benchmarks and three model-harness stacks; results may not generalize to other tasks or models.
- No adversarial or malicious manipulation of skills considered; threat model is purely empirical effects on benign agents.
Open questions / follow-ons
- How can skill authoring processes incorporate grounding and verification support techniques to reduce regressions?
- Can automated detection and mitigation of skill-description osmosis effects be developed to prevent presence-only regressions?
- What skill library designs generalize regression reductions beyond office automation to other domains like coding or dialogue?
- How does repetition or interaction between multiple simultaneous skills affect regression and gain dynamics?
Why it matters for bot defense
Bot-defense and CAPTCHA systems increasingly integrate LLM agents for interaction and automated decision-making. This paper’s decomposition of skill-library impacts provides a cautionary framework for evaluating procedural guidance meant to enhance agent capabilities. Rather than judging added skills solely by aggregate success improvements, practitioners must assess regressions that may degrade performance on critical subtasks, potentially opening new failure vectors exploitable by bots or adversaries. The identification of grounding and verification displacement highlights the importance of carefully designing and verifying input interpretation and output checking when augmenting agents with skills.
In CAPTCHA and bot-defense contexts, procedural skills may serve to guide challenge-solving agents or heuristic filters. Understanding that presence-only skill context (osmosis) can alter behavior without direct invocation suggests an attack surface and robustness concern. Moreover, verification displacement warns that procedural instructions could suppress necessary output validation, risking false acceptances or failures. Thus, comprehensive decomposition and trace-level diagnostic techniques are essential for practitioners to vet skill additions, prioritize auditability, and balance gains against regressions, ensuring reliable automated bot assessments.
Cite
@article{arxiv2607_22520,
title={ The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents },
author={ Darshan Tank and Baran Nama },
journal={arXiv preprint arXiv:2607.22520},
year={ 2026 },
url={https://arxiv.org/abs/2607.22520}
}