Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

Source: arXiv:2607.20911 · Published 2026-07-23 · By Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen, Xiang Fei, Yong Mao, Zihan Xu et al.

TL;DR

Tencent WorkBuddy Bench introduces a comprehensive multi-domain benchmark suite designed to evaluate coding agents in realistic organizational workflows spanning software engineering (Code), front-end web development (Web), office/business workflows (Office), and security operations (Security). Unlike prior public benchmarks that often reuse openly available issues or tutorials—enabling prompt contamination through web-searchable inputs—WorkBuddy Bench constructs contamination-resistant tasks by reverse-engineering real commits, pull requests, or business scenarios and rewriting them into colloquial, underspecified, role-played requests. This prevents direct recovery of prompts from public sources even though the entire dataset (tasks, environments, tests, evaluation harness, and reference solutions) is openly released, enabling full reproducibility and auditability. The benchmark comprises 80 Code tasks, 70 Web tasks, 50 Office tasks, and 60 Security tasks, each scored under a uniform harness but with subset-specific evaluation metrics. The suite furthermore incorporates dataset versioning and canary strings to mitigate contamination over time. Tencent WorkBuddy Bench reports a cross-model leaderboard covering multiple agent families evaluated under two agent harnesses, CodeBuddy Code and Claude Code.

Key findings

• The Code subset contains 80 tasks spanning 5 requester roles and 18 fine-grained categories, with only 10 tasks labeled as bug fixes; the remaining 70 include feature development, code engineering, testing, algorithm engineering, and product analytics.
• Code tasks are admitted only if the baseline workspace scores ≤ 0.3 and an oracle gold patch scores 1.0 on hidden tests, ensuring task difficulty and solution feasibility.
• The Web subset includes 70 tasks organized into 7 categories (page interaction, data visualization, visual design, etc.) and 6 lifecycle modes, with half the tasks requiring artifact generation from scratch and the other half focusing on bug fixes, feature extensions, review, test generation, or format conversion.
• Each subset uses a different scoring regime: Code uses hidden tests, Web uses a rubric combining rule checks, LLM and VLM judges, and agent interaction judges; Office uses a hybrid deterministic rule and evidence-grounded LLM judge; Security uses deterministic scoring scripts.
• The entire benchmark dataset, including task directories, environment images, test suites, evaluation harness, and reference solutions, is openly released, enabling third parties to reproduce and audit every task end to end.
• Cross-model evaluation on two agent harnesses evaluated multiple model families, reporting top Code test scores around 75% and variation across domains (Code, Web, Office, Security) without a combined suite-wide average due to incompatibility of scoring instruments.
• Contamination-resistance is achieved by rewriting original upstream artifacts (commits, CVEs) into underspecified, colloquial requests that withhold root causes and exact references, preventing prompt recovery by web search.
• Dataset versioning and canary strings are used to detect and manage model training exposure over time rather than relying on secrecy.

Threat model

The adversary is a coding agent model potentially trained on publicly available repositories and CVE disclosures and able to query online text to match prompt patterns. The benchmark assumes no adversary can look inside the benchmark’s hidden tests during task solving and cannot recover prompts by direct web search, thanks to rewrite and contamination-resistant task construction. However, adversaries can have partial knowledge of the underlying code or vulnerability details, and exposure over time via dataset release is managed only by versioning, not preven­tion.

Methodology — deep read

1. Threat model & assumptions: The benchmark targets coding agents intended for realistic workplace usage, where adversaries may include models trained or fine-tuned on publicly available code or descriptions, but task prompts must not be trivially recoverable by web search. The benchmark does not claim immunity to models trained on original code or CVE databases but aims to close the searchable-prompt contamination path via rewrite and dataset versioning.

2. Data provenance: Tasks derive from three main sources: (a) real upstream open-source commits/pull requests for the Code subset (34/80 tasks), (b) business scenarios for Web, Office, and synthetic tasks in Code and Security, and (c) CVE-anchored whitebox audit tasks for Security. The Code subset further divides into 'Family A' real OSS tasks, 'Family B' clean-room reimplementations or ports, and 'Family C' fully synthetic workspaces.

The full released dataset contains 80 Code, 70 Web, 50 Office, 60 Security tasks. No raw user data or session logs are used; construction is informed only by internal usage taxonomies and aggregate distributions.

3. Architecture / algorithm: The benchmark is an evaluation suite rather than a model, so the key algorithms relate to task construction and evaluation harness. Tasks use a standardized directory format isolating agent-visible workspace from hidden grading assets, with Docker containers simulating isolated environments.

Evaluation instruments differ per subset:

• Code uses hidden unit tests held out during solve time; gold patches validate oracle solutions.
• Web uses a suite of rule checks (files exist, JSON valid), LLM and vision model judges for semantic and visual correctness, plus agent interaction judges testing state and workflows.
• Office combines deterministic rule checks on documents and workspace state with evidence-grounded LLM judge rubrics.
• Security uses deterministic scoring scripts evaluating vulnerability discovery, reproduction, and analysis.

4. Training regime: N/A for the benchmark itself; evaluation runs leverage two model harnesses, CodeBuddy Code and Claude Code. Tests ensure baseline tasks reject trivial solutions and oracle patches pass fully.

5. Evaluation protocol: Tasks enforce a temporal isolation: the agent sees only the instruction and initial workspace; evaluation assets and tests are introduced only post-solution. Scoring is per task with per-subset metrics; no aggregate across subsets due to differing scoring nature. Admission gates prevent trivial tasks. Results are reported in task-level numeric scores and cross-model leaderboards.

6. Reproducibility: The entire benchmark is released open source with complete task directories, Docker environment images, evaluation harness code, hidden tests, and reference solutions enabling third parties to re-run and audit any task fully. Dataset versions are tagged and periodically refreshed. No user data or search-indexable prompts are included.

Concrete example walkthrough (Code subset): An agent receives a product manager’s request describing a 'checkout-copy experiment' evaluation involving computing conversion and revenue from impression and purchase events. The agent loads the repository in a sandboxed Docker container, explores the relevant modules (across multiple files), writes code changes, and submits a patch. The patch is then evaluated against hidden pytest tests. Only post submission is the test suite run, measuring pass rate to produce a numeric score. The agent must also interpret underspecified prompts (e.g., exclusion windows) by inferring from repository context rather than receiving explicit instructions.

Technical innovations

• A contamination-resistant task construction method that rewrites real commits, pull requests, and business scenarios into underspecified, colloquial, role-played natural language prompts to prevent prompt recovery by web search.
• A unified, multi-domain evaluation suite spanning Code, Web, Office, and Security, capturing complementary organizational workflows within one reproducible harness but with distinct domain-specific scoring instruments.
• An open-release benchmark packaging scheme using containerized sandbox tasks that separate agent-visible workspaces from evaluation assets, enabling full reproducibility, auditability, and contamination resistance without relying on secrecy.
• Oracle-gated tasks with baseline and oracle admission tests to ensure tasks are neither trivial nor unsolvable, guaranteeing meaningful difficulty and solution feasibility.
• A multi-modal Web subset evaluation combining rule-based checks, large language and vision model judges, and agent interaction judges to verify not just static code but dynamic artifact behavior and UI correctness.

Datasets

• Tencent WorkBuddy Bench Dataset — 260 tasks total (80 Code, 70 Web, 50 Office, 60 Security) — openly released on project page with task directories, Docker environments, evaluation harness, tests, and reference solutions

Baselines vs proposed

• Claude Opus 4.8 (Anthropic): Code subset hidden-test score = 74.4%, Web rubric = 68.1%, Office blended = 82.4%, Security deterministic = 64.4%, Overall averaged = 75.0%
• GLM-5.2 (OPEN Zhipu): Code = 71.5%, Web = 67.4%, Office = 79.6%, Security = 76.3%, Overall = 72.9%
• GPT-5.5 (OpenAI): Code = 72.9%, Web = 61.1%, Office = 82.0%, Security = 64.4%, Overall = 72.0%
• HY-3 (Tencent proprietary): Code = 62.9%, Web = 67.7%, Office = 82.1%, Security = 64.5%, Overall = 70.9%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20911.

Fig 1: Tencent WorkBuddy Bench at a glance. Real commits, pull requests, office workflows, and security

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Scores across subsets are not comparable or aggregable due to differing evaluation metrics and modalities; no overall suite-wide average is reported.
• While contamination-resistant prompt construction prevents searchable-prompt leakage, open release implies models trained later may have access to original code or CVEs behind tasks, limiting immunity to exposure.
• Task difficulty leans heavily on navigation and grounding in large repositories rather than pure code synthesis, potentially disadvantaging models with weaker repository-level reasoning.
• Security subset focuses on detection and analysis workflows, not on vulnerability fix synthesis, limiting the scope of security-related agent capabilities assessed.
• Evaluation harness uses two agent backends (CodeBuddy Code and Claude Code), but broader generality across other agents remains untested.
• No explicit adversarial testing or stress evaluation documented to test robustness against crafted or malicious inputs.

Open questions / follow-ons

• How well do agents adapt to the underspecified, real-world-style prompts requiring deep repository navigation and grounding beyond code synthesis?
• How can cross-subset evaluation metrics be harmonized or normalized to produce a meaningful suite-wide aggregate score?
• What is the impact of contamination-resistant task rewrites on agent training when datasets are incorporated into large training corpora over multiple refreshes?
• Can agent performance improve with hybrid multi-modal evaluation signals, and how can this be leveraged for better agent debugging and feedback?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Tencent WorkBuddy Bench offers insights into constructing contamination-resistant tasks that resist obvious leakage paths such as web-searchable prompts. The benchmark’s approach of rewriting real user queries into delivered, underspecified, and role-played instructions can inspire CAPTCHA challenges that require contextual reasoning rather than easy pattern matching or memorization. Its division into multiple real-world domains also highlights that robust bot detection or challenge design must span diverse workflows and verification styles, from deterministic checks to semantic rubrics and interaction-based assessments. The open-source nature and task-directory format further suggest practical approaches to maintaining transparency and reproducibility in adversarial evaluation systems.

Cite

@article{arxiv2607_20911,
  title={ Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction },
  author={ Tencent WorkBuddy Bench Team and Siqi Cai and Shaopeng Chen and Xiang Fei and Yong Mao and Zihan Xu and Zhiheng Lyu and Zhijian Shao and Yuchen Shi and Shuwen Zhang and Chaofan Qiu and Linjie Che and Xiaoxi Zhao and Feng Wu and Kai Zhang and Chaofan Zhu and Yubin Qi and Xiaoyun Liang and Peijie Dong and Yunhao Zhang and Yuanjie Zhu and Ling Jiang and Xianjun Zhang and Zhehang Chu and Anyuan Sang and Zhen Feng and Sen Nie and Shi Wu and Yuanzhen Xu and Xin Li and Ning Yang and Zhiqiang Dong and Hande Dong and Qiang Lin and Yi Liu and Yunsheng Wu and Ke Li and Xing Sun },
  journal={arXiv preprint arXiv:2607.20911},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20911}
}

Read the full paper

• arXiv:2607.20911 (abstract)
• arXiv:2607.20911 (PDF)