Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

Source: arXiv:2607.21482 · Published 2026-07-23 · By Mack Nixon, Liam Wright, Yevgeniya Kovalchuk, Alison Fang-Wei Wu, Martin Danka, Andy Boyd et al.

TL;DR

This paper addresses a key bottleneck in longitudinal health and social science research—data preparation—by evaluating the capabilities of open-weight large language models (LLMs) deployed locally on consumer-grade hardware. Unlike most LLM applications that rely on cloud-based models and thus pose governance risks with sensitive data, this research focuses on fully local deployment, enabling compliance with data privacy restrictions while harnessing AI assistance. The authors introduce RRBench, an open-source evaluation framework containing a curated ground-truth dataset based on six sweeps of the British Next Steps cohort, explicit task definitions involving longitudinal data preparation operations (e.g., category harmonization, multi-wave merging), and automated pipelines to compare LLM-generated R code outputs with human-curated ground truth. Eight recent open-weight LLMs spanning 8B to 35B parameters were benchmarked on 20 data-preparation tasks producing 102 variables. The top models (31-35B parameters) achieved high average task completion rates (up to 87.9%) and correct variable derivations (up to 86.2%), indicating strong potential for local AI-assisted workflows in governance-restricted environments. However, tasks requiring complex survey-specific metadata interpretation remained challenging, and smaller models exhibited silent but material errors in derived datasets affecting downstream analyses.

Key findings

• The best model, Gemma 4 31B, achieved 85.9% balanced variable accuracy and fully completed 75% of the 20 data-preparation tasks on a longitudinal cohort dataset.
• Completeness across models was high (95.4-99.7%), but correctness varied widely (55.1% to 86.2%), indicating errors were mainly in the accuracy of variable derivations rather than omissions.
• Smaller models (e.g., Ministral-3 14B) produced plausible-looking but silently incorrect data that materially biased downstream regression coefficients (e.g., mislabeling NA values reversed directionality of economic activity effects).
• Task complexity measured by prompt length (metadata tokens), script length, and number of wrangling operations negatively correlated with task success (Pearson's r between -0.60 and -0.75).
• Variables with near-universal definitions such as sex, ethnicity, and BMI were consistently well-derived by most models, whereas survey-specific constructs like housing tenure and household income were poorly completed.
• Using less detailed 'lite' prompts decreased model performance by around 5-10% in correctness and task completion, showing prompt specificity matters for LLM success on metadata-rich tasks.
• Downstream linear regression using best model outputs replicated ground-truth coefficients exactly, whereas weaker models produced materially different results due to data errors.
• The open-source SmolAgents framework allows fully autonomous iterative generation, execution, and debugging of R code by LLMs without internet or cloud access.

Threat model

The adversary is implicitly the data governance and privacy threat setting which prohibits uploading or transmitting sensitive individual-level data outside secure, local environments. The system assumes LLMs deployed do not leak or transmit data externally and operate fully offline. No active adversarial manipulation of the LLM or data is considered. Errors arise from model capabilities and prompt design rather than hostile interference.

Methodology — deep read

The authors designed a framework to systematically evaluate open-weight large language models on realistic longitudinal data preparation tasks requiring domain-specific understanding of survey metadata in R.

1. Threat model & assumptions: Researchers operate in governance-restricted Trusted Research Environments (TREs) with sensitive individual-level data; thus, data transmission to external cloud LLM APIs is prohibited. Adversaries are not explicitly modeled since focus is on functional evaluation, not adversarial robustness.

2. Data: The ground-truth dataset is the Next Steps Core dataset, a cleaned, research-ready processed version of a UK longitudinal cohort with six survey sweeps from adolescence to adulthood. It contains a mix of demographic, socioeconomic, health, and education variables, many derived from multiple sweeps. The dataset was curated and validated by experts and stored in wide format. The benchmark consists of 20 pre-defined data-preparation tasks, each with input raw data files, explicit metadata (variable names, labels, value labels in JSON), and human-written R scripts producing an expected output dataset.

3. Architecture/algorithm: The AI agent is implemented via the SmolAgents framework, a lightweight agentic orchestrator compatible with open-weight LLMs lacking native tool-calling. The agent receives a detailed prompt (including task description, structured metadata, and instructions), generates R code to solve the task, executes the generated script locally, then iteratively debugs by regenerating code if errors occur. The input to the LLM includes the context window (~4,000 tokens per task plus prompt template), metadata in JSON, prior execution feedback, and code history. Outputs are R scripts and processed datasets.

4. Training: The evaluated LLMs are pre-trained and released models (8B to 35B parameters, mixture-of-expert or dense architectures) from recent open-weight releases under Apache 2.0. No additional fine-tuning was performed by the authors. Models were run with temperature 0.8 and each task was run 3 times.

5. Evaluation: Performance is measured mainly via three metrics per task: completeness (proportion of expected variables generated), correctness (proportion of generated variables meeting ≥95% cell match or normalized RMSE ≤10⁻4), and balanced performance (product of completeness and correctness). Task-level metrics aggregate variable scores; complete task requires all variables correct. Automated Python scripts compare LLM outputs to ground truth with tolerance for minor differences in missing value coding. Downstream impacts were assessed by comparing regression coefficients on variables derived by LLMs versus ground truth.

6. Reproducibility: The framework (RRBench), ground-truth data processing scripts, evaluation pipelines, and prompts are publicly available on GitHub (https://github.com/UCL-ARC/RRBench). Models evaluated are open-weight and released under Apache 2.0, facilitating community replication and extension.

Example flow: For one task, the agent receives raw survey data files and JSON metadata describing variable labels and value codes. Given a detailed prompt specifying variable derivations (e.g., derive a consolidated sex variable harmonizing multiple sweeps), the model generates an R script performing recoding, merging, and harmonization. The script executes locally to produce a research-ready dataset, which is then compared variable-by-variable to the human-derived ground truth. If errors occur or outputs differ, the agent regenerates code iteratively. Metrics record how many variables were correctly recreated and aggregate to task success.

Technical innovations

• Development of a domain-specific benchmark (RRBench) reflecting complex survey metadata interpretation, multi-wave harmonization, and longitudinal variable derivation in R, addressing gaps in existing benchmarks focused on general software engineering or Python.
• Use of a fully local, agentic LLM workflow (SmolAgents) combining code generation, execution, and iterative debugging within governance-restricted environments, enabling privacy-preserving AI-assisted data preparation.
• Evaluation of open-weight mixture-of-expert and dense LLMs across deployment tiers (consumer to high-end hardware) on realistic longitudinal data tasks, including detailed analyses of prompt design impact on performance.
• Design of automated, quantitative evaluation metrics combining completeness and correctness at variable and task levels, with tolerances for realistic data discrepancies, enabling objective benchmarking of tabular data transformations.
• Open-source release of a portable evaluation framework that can be adapted to other cohort datasets, languages, and data-preparation workflows, promoting transparency and extensibility.

Datasets

• Next Steps Core dataset — approximately 6 survey sweeps of a UK longitudinal cohort with 102 derived variables — cleaned, human-curated dataset stored in wide format (publicly available: https://github.com/cls-data/ns_core)

Baselines vs proposed

• Gemma 4 31B: Balanced variable performance = 85.9%, Complete tasks = 75.0%, Avg task completion = 87.9% vs. Qwen3.6-35B-A3B: 81.7%, 73.3%, 87.6%
• GPT-OSS-20B: 77.8%, 60.0%, 81.0% vs. Devstral-Small-2-24B: 74.2%, 60.0%, 79.7%
• Qwen3.5-35B-A3B: 73.9%, 61.7%, 79.6% vs. Qwen3.5-9B: 62.1%, 53.3%, 68.4%
• Ministral-3-14B-Instruct-2512: 52.6%, 38.3%, 62.0% (lowest correctness and completion among valid outputs)
• Gemma 4 E4B failed to produce any valid output and was omitted from comparison.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21482.

Fig 1: Overview of the evaluation framework. Each (human curated) ground-truth task

Fig 2: Structure of the prompt provided to the AI agent. The prompt consists of two

Fig 3: Task complexity index, inferred from average task completion rate. For each

Fig 4: Task complexity plotted against task success. An individual task consists of the

Fig 5 (page 33).

Fig 6 (page 34).

Fig 7 (page 34).

Fig 8 (page 35).

Limitations

• Evaluation focused on fully autonomous agentic workflows; in practice, human-in-the-loop review is likely needed to catch subtle errors.
• Models struggled with tasks involving complex, bespoke survey metadata and multi-step derivation logic, notably housing tenure and household income.
• Prompt design significantly impacts performance; only a limited set of prompt variations were tested, leaving open optimization space.
• The benchmark uses a single longitudinal cohort dataset (Next Steps), limiting generalizability without extension to other populations or data collection modes.
• No adversarial evaluation or robustness testing against deliberate prompt or data manipulation was performed.
• Performance degradation on long prompts suggests sequence length and metadata complexity are key bottlenecks, not fully addressed.

Open questions / follow-ons

• Can multi-agent architectures decomposing metadata interpretation, code generation, and validation improve performance over a single LLM agent?
• How generalizable are findings beyond the Next Steps cohort to other longitudinal studies with different metadata schemas or programming languages?
• What automated prompt engineering or optimization techniques best enhance LLM performance on complex data-preparation tasks?
• How can silent data errors be detected and mitigated in practice, especially with weaker models or in semi-autonomous workflows?

Why it matters for bot defense

This study offers practical insights for bot-defense and CAPTCHA practitioners exploring the application of open-weight LLMs for sensitive data processing tasks in privacy-constrained environments. It demonstrates that locally deployable large language models can effectively automate complex data preparation workflows, a key bottleneck in longitudinal research datasets, without sending data to the cloud. For CAPTCHA or bot-mitigation systems relying on secure on-device AI capabilities, this shows promise for embedding advanced reasoning and coding agents within restricted environments.

However, the results underscore the importance of understanding metadata complexity, prompt design, and silent error risks when relying on autonomous code generation. Bot-defense engineers should carefully consider the potential for plausible-looking but incorrect outputs that could silently degrade downstream data integrity or analytics. Human-in-the-loop validation and multi-agent orchestration may be necessary mitigations. The open-source evaluation framework can help benchmark and iterate on LLM agent architectures in privacy-preserving settings relevant to bot detection and automated challenge-response scenarios.

Cite

@article{arxiv2607_21482,
  title={ Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks },
  author={ Mack Nixon and Liam Wright and Yevgeniya Kovalchuk and Alison Fang-Wei Wu and Martin Danka and Andy Boyd and David Bann },
  journal={arXiv preprint arXiv:2607.21482},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21482}
}

Read the full paper

• arXiv:2607.21482 (abstract)
• arXiv:2607.21482 (PDF)