SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions

Source: arXiv:2607.15272 · Published 2026-07-16 · By Yasheng Sun, Zezi Zeng, Yifan Yang, Chong Luo, Wenyi Wang, Ziwei Liu et al.

TL;DR

This paper addresses the challenge of automating scientific figure editing guided by natural-language instructions. Scientific figures are complex visual infographics composed of heterogeneous elements such as schematics, plots, photos, and arrows arranged under a tight semantic and stylistic grammar to convey scientific arguments. Editing them usually requires localized compositional operations like relabeling, layout rearrangement, or restyling. The authors introduce SciDiagramEdit, a new benchmark mined from arXiv paper version histories that contains 364 naturally occurring before/after figure pairs annotated with 2,628 atomic editing claims representing real author-driven revision intents. To tackle the diversity of editing operations, they propose a skill-evolution framework: an agent interacts with the editable vector SVG source primitives, executing editing sequences guided by an evolving skill specification distilled from feedback comparing its outputs to author revisions. This iterative learning loop improves the agent's editing accuracy and aesthetic consistency over multiple epochs.

Experiments show the method outperforms strong single-pass baselines including state-of-the-art large language model (LLM) powered SVG editors and raster image regenerators like GPT-IMAGE-2. Their agent achieves the highest semantic faithfulness checklist success rate (93.2%) and instruction-following win rate (75.6%) on the test set, while matching or nearly matching the target's aesthetic measures (UniPercept IAA 47.93 vs 48.25 and ISTA 45.58 vs 46.49). The learned editing skill also transfers well across different backbone LLMs, demonstrating broad generalization. User studies verify superior instruction following and aesthetic appeal compared to top baselines. Overall, this work pioneers learning realistic instruction-driven figure editing from author-revision data at the editable vector level, providing a foundation for future automation tools in scientific diagram editing.

Key findings

• SciDiagramEdit benchmark contains 364 before/after figure pairs from arXiv spanning 23 subjects annotated with 2,628 atomic editing claims.
• Agent trained with skill evolution achieves 93.2% semantic checklist success rate and 75.6% instruction-following win rate on test set, outperforming GPT-IMAGE-2 (88.2%, 77.8%) and AUTOFIGURE-EDIT GPT-5.5 (84.4%, 57.3%).
• On image aesthetics (UniPercept scores), the agent achieves IAA=47.93 and ISTA=45.58, close to human author revisions (IAA=48.25, ISTA=46.49) and surpassing single-pass baselines.
• Skill evolution improves performance across multiple GPT backbones, adding 1.1-5.4% semantic and 3.7-7.6% aesthetic win rate gains.
• Transferring a skill evolved with the strongest GPT-5.5 backbone to other models provides comparable gains, indicating skill generality.
• User studies on 30 samples confirm the proposed agent wins over GPT-IMAGE-2 and AUTOFIGURE-EDIT GPT-5.5 in both aesthetics and instruction following.
• Agent operates on editable vector figure sources (SVGs) enabling local compositional edits and maintaining primitives for inspection and manual co-editing.
• Mining natural paper revisions is an effective supervision source capturing authentic author editing intentions for realistic instruction-driven figure editing.

Methodology — deep read

1. Threat Model & Assumptions: The adversary scenario is not the focus here; rather, the work assumes access to vector source scientific figures and natural language instructions representing real author revision intents. The agent's goal is to produce edited figures matching author revisions in both semantic changes and aesthetics, preserving unrelated content. Adversarial robustness or malicious editing attacks are not addressed.

2. Data: The dataset SciDiagramEdit is mined from arXiv version histories by identifying figure pairs from two versions of the same paper. It contains 364 figure pairs spanning 23 arXiv primary subjects, predominantly machine learning fields. Each pair is annotated with 2,628 atomic editing claims capturing the authors’ own revision intent via natural-language instructions and fine-grained checklists verifying specific edit effects. Figures are decomposed into editable vector SVG primitives plus embedded raster panels using the AUTOFIGURE-EDIT pipeline. Data is split 2:1:3 into training, validation, and test sets.

3. Architecture / Algorithm: The system comprises three main components operating in a loop: (a) Editor (E): an agentic code-writing subprocess controlling an SVG editor by issuing a sequence of calls to manipulate primitives according to the current skill specification S. The Editor loads a skill directory with markdown files describing workflows and tools dictating editing behavior. It has access to CLI utilities for rendering and icon synthesis to assist edits.

(b) Judge (J): a vision-language model scoring the Editor's output on semantic faithfulness and aesthetic preference by comparing the edited figure against the author’s revised figure. Semantic faithfulness is measured via verification accuracy on per-sample checklist questions; aesthetic preference via pairwise VLM comparisons. The scores are combined with an aesthetic gating rule ensuring semantic credit only if aesthetic quality is met.

(c) Coach (C): a code-writing subprocess that reads Editor execution traces, Judge scores, and reference author revisions. It emits patch edits to the skill specification S that improve future Editor performance. This patch-and-merge approach iteratively evolves the skill over minibatches during training.

The skill specification S contains editable markdown files and workflows guiding editing steps, learned rather than hand-crafted. During editing, execution traces of primitive manipulations and feedback signals guide the Coach to synthesize generalizable editing rules capturing author visual grammar.

4. Training Regime: The iterative loop runs for two epochs on the training split. Minibatches of samples are processed by the Editor under the current skill, the Judge scores outputs, and the Coach proposes skill updates accepted if validation metrics improve. A top-K frontier of best skill snapshots is maintained; training returns the best skill found.

5. Evaluation Protocol: Semantic faithfulness is evaluated with the natural-language checklist success rate over atomic editing claims and blind pairwise win rates for instruction following, referencing author revisions. Aesthetic quality is measured with UniPercept’s Image Aesthetic Assessment (IAA) and Image Structure and Texture Assessment (ISTA) scores, plus blind pairwise aesthetic win rates. Ablations test skill effect by comparing no-skill vs evolved skill usage across multiple LLM backbones. Statistical tests details are not specified but metrics are reported on a held-out test set Dtest.

6. Reproducibility: The benchmark dataset and skill specifications are to be publicly released under CC BY-NC 4.0. The underlying vector figures remain subject to arXiv licensing. The framework relies on closed-source GPT-5.X backbones via API with training-time costs. Full training is not reproducible without access to these models. The paper includes sufficient pseudocode and architecture explanations.

Concrete Example End-to-End: Given an input figure SVG and a natural-language instruction to rearrange panels and relabel items, the Editor loads the current skill files describing workflows and utilities. It issues a sequence of SVG manipulations to move vector primitives, insert or edit text labels, and adjust layouts, calling CLI rendering and layout checkers to verify aesthetics. The edited figure is rendered and fed to the Judge along with the original and author target figures to produce semantic and aesthetic scores. The trace of editing calls and scores is passed to the Coach, which generates code patches updating the skill markdown files to improve future editing steps. This loop iterates, progressively refining the editing skill and improving the agent’s accuracy and visual coherence.

Technical innovations

• Mining a novel benchmark of real-world before/after scientific figure edit pairs from arXiv version histories annotated with fine-grained editing claims capturing author intent.
• Agentic skill evolution framework that iteratively distills editing skills from execution traces and author demonstration comparisons into a learnable skill directory guiding an SVG-editing agent.
• Joint Judge model combining semantic fidelity via checklist verification and aesthetic preference through pairwise vision-language comparisons with an aesthetic gating mechanism.
• Operating directly on editable vector figure sources, enabling interpretable local edits and user inspectability, contrasting with prior single-pass raster re-rendering methods.

Datasets

• SciDiagramEdit — 364 before/after figure pairs with 2,628 annotated editing claims — mined from arXiv paper revisions

Baselines vs proposed

• GPT-IMAGE-1.5: semantic success rate = 51.6%, aesthetic IAA = 50.77 vs Ours semantic success = 93.2%, IAA = 47.93
• GPT-IMAGE-2: semantic success rate = 88.2%, aesthetic IAA = 49.6 vs Ours semantic success = 93.2%, IAA = 47.93
• AUTOFIGURE-EDIT (GPT-5.3): semantic success rate = 82.8%, aesthetic IAA = 45.3 vs Ours semantic success = 93.2%, IAA = 47.93
• AUTOFIGURE-EDIT (GPT-5.5): semantic success rate = 84.4%, aesthetic IAA = 44.87 vs Ours semantic success = 93.2%, IAA = 47.93
• Skill evolved on GPT-5.5 transferred to GPT-5.3: semantic win rate improves by +4.7%, aesthetic win rate improves by +8.1%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15272.

Fig 1: Overview of SciDiagramEdit. Left: the Human Revision Repository, naturally occurring before/after

Fig 2: (a) breaks

Fig 3: Overview of the SciDiagramEdit training loop on a representative sample. Given an input figure F in

Fig 4: Qualitative comparison on a representative editing instance. The panels present the input figure, the

Fig 5 (page 5).

Fig 5: Qualitative skill ablation on a representative editing instance. Left: the input figure (top) and the author’s

Fig 7 (page 7).

Fig 8 (page 7).

Limitations

• The benchmark only includes editing instructions explicitly stated in natural-language prompts; it does not cover multi-step reasoning or implicit intent inference from paper context.
• The skill evolution training loop is externally orchestrated and manually supervised rather than fully autonomous online refinement.
• Experiments are conducted at modest scale with a few hundred training pairs and limited evolution steps, leaving open gains possible with larger data or longer training.
• The system depends on closed-weight GPT-5.X models, limiting reproducibility and raising cost barriers.
• Focus on SVG vector edits limits applicability to diagrams that combine raster elements or highly complex graphics without vector decomposition.
• No adversarial robustness or evaluation against malicious editing attempts is explored.

Open questions / follow-ons

• Can this skill-evolution approach scale to larger and more diverse scientific figure edit datasets mined from broader publication sources?
• How might the system incorporate multi-step reasoning over the figure’s scientific argument or infer implicit author revision intents lacking explicit prompt instructions?
• Can the method be extended to jointly edit raster and vector elements or handle more complex multimodal figures blending photos, charts, and schematics?
• What autonomous orchestration strategies could enable fully self-refining skill evolution without human supervision or validation gating?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work demonstrates that natural author revision data can provide rich supervision for training agents to perform complex, compositional edits under natural-language instructions on structured vector graphics. While not directly related to bot-detection or CAPTCHA creation, the skill evolution and agentic editing approach may inspire future semi-automated or interactive CAPTCHA generators producing adaptive vector challenges. The use of fine-grained edit verification with combined semantic and aesthetic evaluation could inform robust evaluation metrics for CAPTCHA puzzles involving graphical manipulations. Additionally, the demonstration of learning transferable procedural skills from natural-language instructions and execution traces may have parallels in evolving adversary or defender agent behaviors in interactive security tasks.

Cite

@article{arxiv2607_15272,
  title={ SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions },
  author={ Yasheng Sun and Zezi Zeng and Yifan Yang and Chong Luo and Wenyi Wang and Ziwei Liu and Jürgen Schmidhuber },
  journal={arXiv preprint arXiv:2607.15272},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15272}
}

Read the full paper

• arXiv:2607.15272 (abstract)
• arXiv:2607.15272 (PDF)