SAGE: Structured Agentic Graph Editing for Software Diagrams
Source: arXiv:2607.01102 · Published 2026-07-01 · By Tyler Sivertsen, Neal Singh, James C. Davis
TL;DR
SAGE presents a hybrid approach to editing software engineering diagrams by combining structured graph representations with language model (LLM) guided natural language editing. The key innovation is the transformation of inherently complex visual diagrams (like Draw.io XML or Mermaid diagrams) into an editable graph model, where user natural language inputs are parsed into structured edit intents. These intents are then deterministically executed with validation and repair steps to maintain structural integrity, versioning, and recoverability. This approach separates semantic interpretation (using AI models) from deterministic, validated graph mutations, overcoming the brittleness of purely image- or text-based diagram generation.
The system supports two workflows: (1) diagram mode where the diagram is parsed into a graph, edited through structured operations derived from natural language edits, and re-serialized; and (2) image mode, supporting masked, prompt-guided raster editing. They evaluate SAGE via a Kubernetes architecture case study, reconstructing a complex diagram and applying multiple edits, measuring correctness, preservation of unrelated elements, and topological consistency. Results show that all prompt-guided structured edits succeeded while preserving unrelated structure and reloading in Draw.io, though layout quality and visual aesthetics had room for improvement. Image editing succeeded in semantic changes but suffered cumulative quality degradation. Overall, SAGE demonstrates the feasibility and advantages of integrating agentic language models with structured, fail-safe diagram editing pipelines.
Key findings
- SAGE reconstructed a Kubernetes cluster architecture diagram as a structured, reloadable editable artifact despite partial layout mismatch.
- All three prompt-guided structured edits (rename node, add monitoring agent nodes, insert authentication layer) were successfully applied to correct targets without disrupting unrelated structure.
- Preservation and topology metrics passed for all edits, ensuring high-level connections remained consistent after modifications.
- Image editing workflow supported semantic edits on raster diagrams but showed degraded visual consistency and layout drift after repeated edits.
- Validation and repair logic prevented invalid Draw.io XML outputs by checking root/layer cells, edges source-target integrity, and repaired common structural issues like missing wrappers.
- Versioned session steps store detailed metadata for each edit, enabling non-destructive editing and recovery of prior diagram states.
- SAGE separates language model reasoning from deterministic graph transformations and validation, improving robustness compared to direct prompt-to-image approaches.
Threat model
Not applicable. The paper focuses on reliable natural language guided editing of software diagrams rather than adversarial security threats. The system trusts the user inputs and language model interpretations, emphasizing structural validation to prevent corrupted or invalid diagram states.
Methodology — deep read
Threat model & assumptions: The adversary is not formally defined as this is a tooling paper focused on reliability in natural language to graph editing for diagrams. The system assumes users issue natural language edits to a diagram with intent to maintain structural correctness.
Data: The primary evaluation data is synthetic prompts reconstructing a Kubernetes architecture diagram with labeled components and connections. The diagram is imported and reconstructed into SAGE's internal editable DiagramModel representation. No large-scale datasets are used; evaluation is mainly case study and unit-test based.
Architecture/Algorithm: SAGE parses Draw.io XML into a DiagramModel graph composed of nodes, edges, labels, style, and geometric info. Natural language prompts are converted into ParsedEditIntent objects encoding operation type (e.g., rename, insert-between), target selectors, replacement content, and layout constraints. These intents are then analyzed into concrete graph operations (label updates, node insertion, edge reconnection). Validation checks preconditions and Draw.io XML structural correctness, applying repair logic for common issues like missing root elements. The pipeline separates model-driven reasoning (intent parsing, target resolution) from deterministic graph edits and validation to reduce brittleness.
Training regime: Not applicable. The system uses pre-trained language models (e.g., OpenAI and Gemini) only for interpreting natural language inputs into structured intents. Model outputs are constrained and filtered by application logic.
Evaluation protocol: They reconstruct a Kubernetes diagram from a natural language prompt to form a structured baseline. Then apply three prompt-guided edits via the prompt-edit interface. They measure four metrics: structural validity (whether the edited diagram serialized to valid Draw.io XML and reloads), edit correctness (intended change applied to correct targets), preservation (unrelated elements unchanged), and topology checks (expected high-level connections remain intact). A run is successful only if all hold simultaneously. The image editing workflow is evaluated separately for semantic applicability and degradation.
Reproducibility: Source code is released on GitHub; diagrams and evaluation artifacts are available. The paper includes a demo video. The system depends on external LLM APIs. While unit tests and case study are detailed, no large-scale benchmark or cross-validation is reported.
Technical innovations
- Representation of software diagrams as editable graph models with explicit nodes, edges, labels, and geometry reconstructed from Draw.io XML.
- Parsing of natural language edit instructions into structured ParsedEditIntent objects that capture operation types, targets, and constraints, separating semantic interpretation from execution.
- A transformation pipeline that converts edit intents into deterministic graph operations with validation and repair to maintain valid serialized artifacts.
- Versioned session artifact management that records each edit step's prompt, model interpretation, execution trace, and result for recoverability and inspection.
- Dual editing workflows: structured prompt-guided diagram editing with correctness guarantees and secondary mask-based image editing for flexible visual adjustments.
Datasets
- Kubernetes architecture diagram case study — 1 synthetic diagram reconstructed and edited interactively — public from Kubernetes docs (https://kubernetes.io/docs/concepts/architecture/)
Baselines vs proposed
- Reconstructed baseline diagram: partial correctness (layout differed), successfully reloadable vs prompt-guided edits: 100% success in correctness, preservation, and topology (Table 1)
- Image editing workflow: semantic edits applied successfully but showed visual degradation after repeated edits vs structured editing: structured editing preserved connectivity and artifact validity better
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.01102.
Fig 1: Diagram-editing workflow in SAGE.
Fig 2: Kubernetes case study artifacts: the reference diagram, final structured diagram-editing result, and final image-editing
Fig 3 (page 4).
Fig 4 (page 4).
Limitations
- Layout quality is only partially addressed; reconstructed diagrams often have overlapping or aesthetically suboptimal layouts requiring manual adjustment.
- Semantic validation beyond basic XML structure (e.g., domain-specific architectural invariants, full group membership) is incomplete or not implemented.
- Evaluation is limited to a single case study diagram and a small set of edit prompts; broader benchmark and adversarial inputs are not tested.
- Image editing workflow lacks explicit graph representation and validation, resulting in cumulative visual degradation after multiple edits.
- Transactional protection for aborting invalid edits during active sessions is noted as future work rather than implemented.
- Interpretation and transformation remain partly dependent on language model outputs, which could introduce ambiguity or errors in complex scenarios.
Open questions / follow-ons
- How to incorporate semantic architectural constraints or domain-specific validation to enhance correctness beyond basic graph structure?
- How to improve automated layout optimization to generate visually clear, non-overlapping diagram arrangements that preserve user intent?
- How well does SAGE generalize to different diagram types, larger-scale systems, or collaborative multi-user editing workflows?
- Can the system be extended to handle adversarial or ambiguous natural language instructions more robustly, possibly with interactive disambiguation?
Why it matters for bot defense
Bot-defense engineers and CAPTCHA practitioners can draw parallels from SAGE's approach in treating complex graphical artifacts as formally structured objects rather than static images for editing or validation. The separation of AI-driven interpretation from deterministic, validated state transformations enhances robustness and failure detection, a principle useful in designing bot-detection workflows where ambiguous inputs require safe handling. Furthermore, versioned, recoverable artifacts allow auditability and rollback, important for forensic tracking in security-sensitive interactions.
While CAPTCHAs do not directly involve diagram editing, the methodology of combining language or image model outputs with structured validation and repair can inspire more reliable challenge generation and verification mechanisms. The layered approach—decoupling semantic understanding from execution—can help bots and legitimate user workflows be better distinguished and controlled by maintaining explicit intermediate representations rather than opaque end results.
Cite
@article{arxiv2607_01102,
title={ SAGE: Structured Agentic Graph Editing for Software Diagrams },
author={ Tyler Sivertsen and Neal Singh and James C. Davis },
journal={arXiv preprint arXiv:2607.01102},
year={ 2026 },
url={https://arxiv.org/abs/2607.01102}
}