MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

Source: arXiv:2606.16748 · Published 2026-06-15 · By Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

TL;DR

MYPCBENCH addresses a significant gap in evaluating personal assistant agents for computers: current benchmarks test agents in impersonal, empty environments, lacking persistent user data, context, and cross-application consistency that real assistants require. It introduces a reproducible Linux desktop environment seeded with a canonical persona, Michael Scott, containing 17 pre-logged-in web apps and full desktop software, populated with thousands of realistic records spanning banking, calendar, chat, travel, orders, and browsing. The benchmark defines 184 real-world inspired tasks derived from a large personal-assistant user community, emphasizing multi-application, history-dependent challenges. Six state-of-the-art closed- and open-weight models are evaluated uniformly using a computer+bash action interface and a rubric-based LLM judge.

The results reveal that the best agent, Claude Opus 4.6, completes only 55.4% of tasks perfectly, with performance degrading sharply as tasks span more applications or longer trajectories. Other models fall far behind—GPT-5.5 achieves 29.3% perfect overall but just 4.5% on tasks touching 7+ apps, while open-weight models like Qwen 3.5 perform worse and collapse under multi-tool complexity. Failures primarily arise from premature task termination, skipping required applications, surface errors, partial completions, and hallucinated persona data. MYPCBENCH provides a rich, reproducible framework to drive research toward genuinely personalized, persistent, multi-application computer-use assistants.

Key findings

• Claude Opus 4.6 fully solves 55.4% of the 184 MYPCBENCH tasks, the only model above 50%.
• Claude Sonnet 4.6 achieves 39.1% perfect task completion; GPT-5.5 scores 29.3%; GPT-5.4 mini 19.0%.
• Open-weight Qwen 3.5 35B-A3B and 9B models reach only 7.6% and 2.7% perfect respectively, with large performance drops when interacting via dual-tool interfaces.
• Tasks touching 7 or more applications are notably challenging: Opus 4.6 solves only 36% perfectly, GPT-5.5 4.5%, and other models near 0%.
• Trajectory Efficiency metric shows Opus 4.6 extracts 3.61 rubric points per agent step, over 5x the 0.65 points by Qwen 9B, highlighting efficiency gaps.
• Failures cluster mainly in premature task completion (354 rubric failures), skipping required apps (323 failures), UI surface errors (129), partial artifact completions (47), and hallucinated personal data (31).
• Personalization-heavy task types like multi-step orchestration, aggregation & reporting, pattern inference, and cross-source reconciliation show the largest gaps, with Opus achieving 82% perfect on pattern inference vs 45% for GPT-5.5.
• Model performance scales negatively with trajectory length—Opus keeps improving up to the 100-step limit, while GPT-5.5 and Qwen saturate earlier.

Threat model

n/a — The work does not explicitly define an adversarial threat model since the focus is on benchmarking agent capabilities in personalized desktop environments, not on security or attack resistance. The implicit assumption is that the agent operates within a trusted, consistent user environment seeded by a single persona and must reliably act without external data leakage or cheating.

Methodology — deep read

The paper proposes a new benchmark, MYPCBENCH, for evaluating personal assistant agents on a fully-seeded Linux desktop environment representing a single consistent user (a canonical persona, Michael Scott).

1. Threat Model and Assumptions: The adversary analogy is not explicit, but the benchmark assumes agents must act over a complex, persistent user environment with cross-application contextual coherence and personal data. The agent cannot cheat by accessing data outside the seeded environment. The personal identity and data are consistent and embedded end-to-end.

2. Data: The environment seeds 17 web applications modeled as local Next.js clones of real consumer services (banking, calendar, travel, messaging, food delivery, ecommerce, and more) plus LibreOffice and Firefox with realistic browsing profile. It contains 1,812 bank transactions, 2,398 emails, 679 calendar events (many recurrent), 2,526 messages, multiple retail & rideshare orders, 10,746 web visits, etc. Data seeding is deterministic from a JSON persona spec. The data is cross-linked (booking a trip also creates bank charges, calendar events, messages, etc.) supporting personalization across apps.

3. Architecture/Algorithm: The benchmark is agnostic to agent type but uses a uniform interface describing a standard computer-use API surface: mouse and keyboard actions supplemented with a bash shell tool. Six commercial and open-weight large language model agents are benchmarked: Claude Opus 4.6 and Sonnet 4.6, GPT-5.5 and 5.4 mini, Qwen 3.5 35B-A3B and 9B. A thin harness drives each agent via the OSWorld HTTP Control API to issue actions and receive full desktop screenshots.

4. Training Regime: Training the LLMs is not part of this work. Benchmark evaluation runs each model on all 184 tasks with a uniform 100-turn LLM call budget per task, with fresh VM snapshot resets between tasks. Each agent receives the same contextual persona data.

5. Evaluation Protocol: Tasks are derived from 2,749 real user requests trimmed and adapted to fit the seeded environment. Tasks cover six behavioral types (bounded action, multi-step orchestration, cross-source reconciliation, aggregation & reporting, personal lookup, pattern inference). Each task includes a natural-language rubric with multiple criteria (mean 6.5 criteria per task) used with an LLM judge (gemini-3.1-flash-lite-preview) that ingests the full agent trajectory (screenshots + actions) and outputs pass/fail per criterion. Metrics reported include Perfect Task Completion (all rubric criteria pass), Rubric Score (weighted partial credit), and Trajectory Efficiency (rubric points gained per agent step).

6. Reproducibility: The environment is publicly released as a Docker-wrapped QEMU Ubuntu 24.04 VM with seeded persona data. Source code for tasks, rubrics, harness, and judge are open. The full data generation pipeline is deterministic from the persona spec, allowing replication and extension.

Detailed Example: One analyzed trajectory is a complex Dundies-lifecycle plan task spanning 10 apps with 9 rubric criteria, successfully completed by Claude Opus in 99 steps. Screenshots show the sequential use of apps such as calendar, messaging, browser, and spreadsheets, illustrating cross-app state maintenance and context tracking required for personalized coordination.

Fail Mode Analysis: Agent failures are categorized into premature DONE signal, skipping required applications, surface-level UI errors, partial completions, and hallucinated persona data. Different model families exhibited distinct failure modes, e.g., GPT models often ended prematurely, Qwen models hallucinated persona data, and Claude models leveraged bash scripting shortcuts.

Technical innovations

• Creation of a fully seeded, cross-application, consistent, and reproducible desktop environment representing a canonical user persona for personalized agent evaluation, unlike previous impersonally seeded benchmarks.
• Integration of 17 realistic, pre-logged-in web application clones plus a full desktop Linux stack, enabling naturalistic multi-application personalization tasks.
• A large-scale, richly annotated task suite of 184 real-world inspired user requests drawn from a prominent personal assistant user community (OpenClaw), each with detailed multi-criteria natural language rubrics.
• Use of an LLM-powered rubric judge that grades full-trajectory agent executions with step-level screenshots and actions for reliable partial and perfect credit evaluation.
• Standardization of a uniform computer+bash action surface across six powerful large language model agents, enabling apples-to-apples comparative benchmarking at scale.

Datasets

• MYPCBENCH environment data — 42,000 user-facing records (bank transactions, emails, calendar events, messages, orders, browsing history) — synthesized from Michael Scott persona
• Task set — 184 personalized, multi-application natural language tasks — drawn and adapted from 2,749 anonymized real OpenClaw user requests

Baselines vs proposed

• Claude Opus 4.6: perfect task completion = 55.4% vs Claude Sonnet 4.6: 39.1%
• Claude Opus 4.6: rubric score = 81.8% vs GPT-5.5: 54.1%
• GPT-5.5: perfect task completion = 29.3% vs GPT-5.4 mini: 19.0%
• Qwen 3.5 35B-A3B: perfect task completion = 7.6% vs Qwen 3.5 9B: 2.7%
• At 7+ applications per task: Claude Opus 4.6 perfect = 36%, GPT-5.5 = 4.5%, others ≈ 0%
• Trajectory efficiency (rubric points per step): Opus 4.6 = 3.61 vs Qwen 3.5 9B = 0.65

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.16748.

Fig 1: Overview of MYPCBENCH. A reproducible Linux-desktop benchmark for personally intelligent

Fig 2: MYPCBENCH environment suite. The 17 logged-in web apps span six SimilarWeb top-

Fig 3: Left: per-task-type perfect rate (%), models × task types (all models with computer+bash).

Fig 4: One full successful Opus trajectory, the Dundies-lifecycle plan on long_horizon-f050

Fig 8: The MYPCBENCH task-review interface used during quality assurance. The left pane

Fig 10: Trajectory vignettes, part 1 of 2. Rows: Claude Opus 4.6 PASS on aggregation-f001,

Fig 11: Trajectory vignettes, part 2 of 2 (continued from Figure 10). Rows: GPT-5.5 FAIL

Limitations

• Only a single canonical persona (Michael Scott) is seeded, limiting evaluation of diverse user behaviors and data distributions.
• Evaluation is performed on a carefully simulated but fixed desktop environment rather than live or dynamically changing personal computers.
• The 100-turn budget per task may constrain some agents on longer or more complex trajectories, potentially truncating performance.
• Analysis does not explore adversarial or actively hostile user/environment scenarios against agents.
• Agents are evaluated with fixed tool surfaces; future models with different toolsets might need adaptation to integrate with MYPCBENCH.
• The paper does not report human user studies or deployment analysis to validate real-world utility beyond benchmark metrics.

Open questions / follow-ons

• How do different user personas, with varying behaviors and data complexities, impact agent performance and failure modes?
• Can agents adapt online to evolving user data or correct errors across multi-application workflows within this benchmark?
• What specific architectural or training improvements can reduce premature task termination and hallucinated persona data failures?
• How would human-in-the-loop feedback or interactive correction influence agent success on long-horizon multi-step tasks?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, MYPCBENCH highlights the complexity and challenges of developing agents that can operate as truly personalized desktop assistants over persistent, multifaceted user data and sessions. Evaluations focusing solely on isolated, stateless, or impersonal tasks may overlook critical failure modes arising from realistic user data diversity and multi-application coordination. The benchmark's findings suggest that attackers or automated systems aiming to simulate human computer use must handle long trajectories, cross-app consistency, and personalized context to avoid detection or produce convincing behavior.

Practitioners designing bot-defense mechanisms or CAPTCHAs could learn from MYPCBENCH’s structured evaluation of personalization challenges. For example, detection features could exploit agents' common failure modes such as premature task termination or hallucination of user data. Additionally, the benchmark’s environment and tasks could serve as a testbed for evaluating how well security mechanisms resist sophisticated personalized automated interactions that leverage agent coding and bash scripting shortcuts.

Cite

@article{arxiv2606_16748,
  title={ MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents },
  author={ Lawrence Keunho Jang and Andrew Keunwoo Jang and Jing Yu Koh and Ruslan Salakhutdinov },
  journal={arXiv preprint arXiv:2606.16748},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.16748}
}

Read the full paper

• arXiv:2606.16748 (abstract)
• arXiv:2606.16748 (PDF)