PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

Source: arXiv:2606.31154 · Published 2026-06-30 · By Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen et al.

TL;DR

The PPT-EVAL benchmark addresses the challenge of evaluating AI agents on realistic, complex PowerPoint slide creation and editing tasks. Unlike existing benchmarks that focus on programmatic API interactions or text-to-slide generation, PPT-EVAL enables full GUI-level interaction with PowerPoint Online, covering a rich set of multimodal slide features such as layouts, animations, transitions, SmartArt, and advanced graphics. The benchmark comprises 120 carefully curated tasks spanning easy, medium, and hard difficulty levels over 12 openly licensed PowerPoint decks representing diverse real-world presentation styles and domains. A key novel contribution is its rubric-based evaluation framework, which provides detailed partial credit scoring, penalizes extraneous edits, and generates interpretable natural language feedback. This method achieves strong correlation (Kendall’s τb=0.77) with human judgments, overcoming limitations of binary success metrics and better capturing agent progress on complex multimodal goals. Evaluation of current state-of-the-art computer-use agents reveals substantial performance gaps relative to humans, highlighting significant headroom in GUI-based agent capabilities on realistic presentation-editing workflows.

Key findings

• PPT-EVAL includes 120 PowerPoint tasks over 12 files, stratified into easy (51), medium (39), and hard (30) tasks.
• Rubric-based evaluation achieved a Kendall’s τb correlation of 0.77 and Spearman’s ρ of 0.84 with human scores.
• Strong closed models like Claude-4.5-Opus achieve only 45% Success Rate and 0.57 average rubric score, far below human baselines of 80% success and 0.90 average score.
• GUI-based agents significantly underperform compared to API/CLI-based agents despite having richer access to PowerPoint features.
• Open-weights models (OpenCUA, QWEN3-VL) have lower success rates (14%-28%) and partial scores (0.23-0.42), with larger models generally performing better.
• Tasks involving advanced PowerPoint features like tables, animations, and non-standard layouts are more challenging for agents than text or shape editing.
• Partial credit rubrics better capture meaningful progress than binary metrics which often miss nuances of partial task completion.
• Rubric construction required ∼150 human hours to refine LLM-generated scoring trees and scoring functions.

Threat model

The adversary is an AI computer-use agent attempting to complete natural language PowerPoint slide editing tasks using only GUI-level interactions within a sandboxed PowerPoint Online environment. The agent cannot directly manipulate source files or APIs beyond the GUI, nor can it see human evaluations or obtain privileged information about scoring rubrics during task execution. The evaluation framework assumes agents may only partially complete tasks and rewards accordingly, without assuming perfect solutions.

Methodology — deep read

1. Threat Model & Assumptions: The benchmark assumes a realistic threat model of AI computer-use agents tasked with performing PowerPoint slide editing via GUI-level interaction. The adversary is essentially the agent, which must understand natural language task instructions and manipulate slide content accurately using the full capabilities of PowerPoint Online's web interface. There is no assumption that the agent has direct API or internal file format access beyond the GUI interaction. The evaluation is designed to capture partial success without assuming perfect task completion.

2. Data: Tasks were derived from 12 openly licensed PowerPoint decks sourced from the Internet Archive encompassing diverse subjects, slide styles, and multimodal content such as images, tables, animations, and non-standard layouts. The decks contain 404 unique slides. A semi-automatic pipeline using an exploratory Claude-4-Sonnet agent generated 471 initial candidate tasks which were filtered to 120 final tasks (10 per file) by six human annotators for clarity, usefulness, and feasibility. Tasks are categorized as easy, medium, or hard based on anticipated user effort and complexity.

3. Architecture/Algorithm: The evaluation framework centers on a hierarchical, tree-structured rubric system inspired by Mind2Web 2. Each task rubric decomposes into critical and non-critical evaluation criteria, with leaf nodes implementing scoring functions that assess various aspects from content presence to layout correctness and visual fidelity. Scoring incorporates partial credit for intermediate progress, penalties for extraneous or unintended changes, and natural language explanations synthesized by LLMs from leaf node feedback. The rubric aggregation formula departs from strict gate-then-average rules to a softened penalty scheme allowing meaningful partial credit while maintaining the importance of critical criteria.

4. Training Regime: Not applicable as this is a benchmarking and evaluation dataset/framework paper. However, the evaluation uses proprietary and open-weights computer-use models (Claude-4.5-Opus, Claude-4-Sonnet, OpenCUA variants, QWEN3-VL) executed via a sandboxed Ubuntu environment running PowerPoint Online in Chromium browsers. Agents interact via GUI-level interfaces simulating mouse, keyboard, and scrolling actions. Each task is allotted a 30-step budget with concurrency set to 3 parallel threads. LLM and VLM calls for rubric evaluation are made using Claude-4-Sonnet.

5. Evaluation Protocol: Two main metrics are defined: Success Rate (percentage of tasks with perfect rubric score 1) and Average Score (mean rubric score accounting for partial credit). Benchmarking is performed over the 120 tasks, with results reported overall and broken down by difficulty and task type (e.g., shapes, tables, animations). A human baseline from five participants with mixed PowerPoint skills is also collected. A meta-evaluation study samples 30 tasks and compares rubric scores against human annotations on tasks with zero, some, significant, and perfect progress, demonstrating strong rubric-human agreement. Variance and qualitative analyses of rubrics are provided.

6. Reproducibility: The benchmark tasks, rubrics, sandbox environment, and evaluation protocols are publicly released at https://microsoft.github.io/ppteval. While proprietary models are evaluated closed-source, open-weights baselines (OpenCUA, QWEN3-VL) are also benchmarked to enable replication and extension by the community. Scripts for uploading tasks, running sandboxes, and scoring are included. The rubric construction involved extensive human refinement of LLM-generated drafts.

Example end-to-end: For a typical task, a fresh PowerPoint Online session is launched with the original deck. The agent receives a natural language instruction (e.g., add a fourth vCPU circle and update a diagram). The agent interacts via simulated mouse and keyboard actions to edit the deck. After the task completes or step budget expires, the modified deck is saved. The rubric’s tree evaluation functions compare the original and modified files, screenshots, and extracted metadata to compute partial scores and generate explanation texts. These aggregate into an overall rubric score reflecting the degree of task completion and error penalty.

Technical innovations

• Introduction of a comprehensive PowerPoint GUI-level interaction benchmark covering multimodal, rich slide editing tasks beyond prior API-focused benchmarks.
• Design of hierarchical, tree-structured rubrics that allocate partial credit, penalize extraneous edits, and incorporate LLM/VLM assessments for semantic and visual correctness.
• A modified aggregation formula for rubric scoring that balances critical and non-critical criteria to reward partial task completion better than prior gate-based methods.
• A semi-automatic rubric construction pipeline combining LLM-generated draft structures with extensive human expert refinement to produce reliable, interpretable scoring.
• Use of sandboxed PowerPoint Online environments with simulated mouse/keyboard actions enabling realistic, end-to-end GUI interaction benchmarking without Microsoft 365 subscription.

Datasets

• PPT-EVAL Benchmark — 120 tasks across 12 openly licensed PowerPoint decks (~404 slides total) — sourced from Internet Archive

Baselines vs proposed

• Human Baseline: Success Rate = 80%, Avg. Score = 0.90
• API Baseline (Claude-4.5-Opus with pptx skill): Success Rate = 62%, Avg. Score = 0.81
• Claude-4.5-Opus (GUI): Success Rate = 45%, Avg. Score = 0.57
• Claude-4-Sonnet: Success Rate = 42%, Avg. Score = 0.53
• Computer-Use-Preview: Success Rate = 38%, Avg. Score = 0.49
• OpenCUA-32B: Success Rate = 28%, Avg. Score = 0.42
• OpenCUA-7B: Success Rate = 24%, Avg. Score = 0.36
• QWEN3-VL-8B: Success Rate = 16%, Avg. Score = 0.27
• QWEN3-VL-32B: Success Rate = 14%, Avg. Score = 0.23

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31154.

Fig 1: Illustration of task-solving in the PPT-EVAL Benchmark. Given a task, an agent interacts with a file in a sandboxed instance of

Fig 2: Task setup and evaluation workflow.

Fig 3: shows examples of slides across the files and Fig. 4

Fig 4: Distribution of elements in slides relevant to PPTArena’s

Fig 5: shows the task distribution by difficulty. Examples

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Benchmark tasks and rubrics focus on PowerPoint Online’s web version and may not generalize to desktop or other presentation software.
• The rubric evaluation depends on LLM and VLM calls which can occasionally hallucinate or produce inconsistent judgments.
• Human effort required to create and refine rubrics was substantial (~150 hours), posing a scalability challenge for expanding tasks.
• Current benchmark lacks adversarial or out-of-distribution evaluation scenarios to test agent robustness under attack or domain shifts.
• GUI-based agents evaluated remain largely immature, indicating that benchmarking results primarily expose current model limitations rather than established capabilities.
• Subjective aesthetics and cultural bias may influence scoring despite rubric attempts to penalize poor style, which could limit evaluation fairness.

Open questions / follow-ons

• How can rubric-based evaluation be further automated or scaled with less human effort while maintaining reliability?
• What are effective training paradigms to improve GUI-based agents’ proficiency on complex multimodal slide editing tasks?
• How do these agents perform under adversarial task modifications or more dynamic, interactive slide editing workflows?
• Can the rubric framework be generalized to other GUI-intensive productivity applications beyond PowerPoint?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the PPT-EVAL benchmark underscores the challenges of evaluating and improving AI agents operating in rich, multimodal GUI environments with complex task goals. The detailed rubric-based evaluation framework provides a compelling approach to capture partial success and agent errors beyond binary metrics, enabling more nuanced assessment of agent capabilities. This principle could inspire enhanced evaluation methods for CAPTCHA-solving or bot-detection systems where tasks involve partial completion and multimodal interaction. Additionally, the paper’s exploration of agent weaknesses on advanced GUI tasks highlights attack surfaces and automation limits relevant to bot mitigation strategies in productivity applications. Understanding such benchmark results helps engineers recognize emerging AI strengths and vulnerabilities in real-world GUI manipulation scenarios.

Cite

@article{arxiv2606_31154,
  title={ PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks },
  author={ Apurva Gandhi and Vishwas Suryanarayanan and Raja Hasnain Anwar and Firoz Shaik and Shubhang Desai and Thong Q. Nguyen and Muhammad Taqi Raza and Vishal Chowdhary and Graham Neubig },
  journal={arXiv preprint arXiv:2606.31154},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31154}
}

Read the full paper

• arXiv:2606.31154 (abstract)
• arXiv:2606.31154 (PDF)