PowerAgentBench-Dyn: A Benchmark for Agentic AI in Power System Dynamic Studies
Source: arXiv:2606.20401 · Published 2026-06-18 · By Qian Zhang, Andrea Pomarico, Costas Mylonas, Magda Foti, Alberto Berizzi, Le Xie
TL;DR
This paper addresses the challenge of applying agentic AI based on large language models (LLMs) to power system dynamic studies—an important yet complex engineering domain that involves multi-step workflows, iterative simulation, and domain expertise. The authors propose PowerAgentBench-Dyn, a novel benchmark framework designed to evaluate AI agents on tasks that cannot be solved by a single-shot computation or deterministic script, but require multi-round tool use, reasoning, and constrained action-taking akin to experienced power engineers. Two initial tasks are presented: (1) Dynamic Model Quality Review, where agents iteratively run model validation tests, diagnose failures, and propose constrained edits to dynamic models; and (2) Dynamic Security Risk Screening, where agents leverage semantic memory and a limited simulation budget to rank contingencies by risk and assess mitigation options. The benchmark provides simulation environments, action/observation definitions, and metrics emphasizing outcome reliability, constraint compliance, and process efficiency. Results from tests on Anthropic and other LLMs demonstrate distinctive agent behaviors and highlight reproducibility challenges due to stochasticity. The benchmark offers a reproducible and extensible foundation for future research integrating agentic AI into power system dynamic operation and planning workflows.
Key findings
- In the Dynamic Model Quality Review (DMQ) task using a modified WECC solar model, Anthropic Opus 4.8 and Sonnet 4.6 repaired the dynamic model successfully in 10/10 runs, typically requiring only 1 iteration to reduce controller gains by approximately an order of magnitude.
- Haiku 4.5 succeeded in 9/10 runs with a median of 2 iterations, showing less efficient convergence and occasional failure to converge within the 5 iteration budget.
- In the Dynamic Security Risk Screening (DSR) task, Opus 4.7, Sonnet 4.6, Gemini 3.1 Pro (Google), and GPT-5.5 (OpenAI) achieved 90-100% workflow success rates across 10 independent runs identifying and simulating top-risk contingencies correctly.
- Haiku 4.5 completed 70% of DSR workflows; Qwen 7B failed to complete due mainly to tool execution issues rather than reasoning errors.
- Severity score estimates and voltage extrema predicted by successful agents closely matched ground truth values, demonstrating effective KPI estimation.
- Synchronous Condenser (SC) mitigation evaluation showed all successful agents consistently identified that SC installation at Bus 26 was critical to stabilize a key contingency (L02-25@160 ms).
- Median completion times for DSR tasks ranged from ~5:43 minutes (GPT-5.5) to ~11:47 minutes (Haiku 4.5), balancing accuracy with operational speed.
- The benchmark enforces constraint compliance such as limiting parameter edits and respecting locked data files; agents receive credit only if fully compliant.
Threat model
The benchmark assumes a non-adversarial AI agent acting within predefined constraints such as locked model files, parameter edit budgets, and simulator interfaces; the adversary is effectively the AI system limited by these operational guardrails. There is no explicit attacker trying to subvert or exploit the system.
Methodology — deep read
The authors design PowerAgentBench-Dyn to simulate realistic power system dynamic study workflows that demand iterative decision making under constraints, reasoning, and engineering judgment beyond static classification or supervised learning.
The threat model assumes an AI agent acting autonomously but restricted by allowable actions, model file edit constraints, and simulation budgets. The adversary is a benign AI user agent; no explicit security adversary scenario is modeled.
Data consists of publicly available or modified standard dynamic power system models, such as the WECC solar model for DMQ and synthetic IEEE 9-bus or 39-bus systems for DSR. Input case files, dynamic model definitions (.dyr), test configurations, semantic memory contents, and candidate fault datasets are released for reproducibility. Confidential industry data constraints are noted.
The DMQ benchmark exposes agents to an interactive environment with observations (simulation outputs, waveform diagnostics), a constrained action space (limited to four REECAU1 controller gains adjustments within five iterations), simulator tools (open-source DMView + PSS/E), user constraints (locked files, action budgets), and evaluation metrics including full-suite pass rates, waveform fit, LVRT recovery, and compliance.
The DSR benchmark interfaces LLM agents with DIgSILENT PowerFactory via the Model Context Protocol (MCP) framework. Observation space includes fault summaries, semantic memory queries, and KPI estimates. Actions cover querying memory, selecting contingencies for simulation within budget, and evaluating mitigation measures (e.g., Synchronous Condenser installations) under constrained resources. Success metrics include top-k ranking overlap with ground truth, severity score errors, mitigation success rates, runtime, and constraint adherence.
Agents from Anthropic (Opus, Sonnet, Haiku), Google (Gemini), OpenAI (GPT-5.5), and Alibaba Cloud (Qwen 7B) are evaluated over 10 independent stochastic runs per task. The evaluation records success rates, median parameter adjustments or iterations, KPI prediction errors, and timing.
The benchmark fixes simulator versions, seed settings, model files, and scoring protocols to ensure deterministic task reproducibility, while agent output stochasticity is embraced and reported through repeated trials. The authors note the challenge of reproducible semantic memory interactions and tool execution failures affecting some runs.
Each task is carefully crafted to match real engineer workflows emphasizing multi-round tool usage, constrained parameter modification, interpretation of simulation outputs, and report generation. Agents interact through APIs exposing domain-specific simulation tool functions rather than direct coding or static prediction.
Overall, the methodology centers on embedding LLM agents into closed-loop dynamic study ecosystems constrained by real-world engineering practices and measuring their ability to autonomously plan, act, interpret, and comply with rigorous standards.
Technical innovations
- Formulation of a novel agentic AI benchmark framework focused on iterative, constrained, and tool-grounded dynamic power system studies, differing from conventional static or supervised tasks.
- Development of the Dynamic Model Quality Review task requiring multi-step model validation, diagnosis, and constrained parameter updates with enforced compliance and auditability.
- Creation of the Dynamic Security Risk Screening task integrating semantic memory queries, limited simulation budgets, KPI-based contingency ranking, and mitigation evaluation in a controlled simulator-LLM loop.
- Implementation of a deterministic task and simulator environment enabling reproducible metric-based evaluation despite stochastic agent policies and multi-round tool interactions.
Datasets
- WECC modified solar dynamic model — size unspecified — publicly available with modified parameters provided in the benchmark
- Unseen fault dataset for short-circuit contingencies — 10 distinct contingencies (bus and line faults with defined durations) — included in released benchmark package
- Semantic memory datasets of prior simulation cases and results — size unspecified — included in open benchmark repository
Baselines vs proposed
- Haiku 4.5 success rate on DMQ task: 90% vs Opus 4.8 and Sonnet 4.6: 100%
- Workflow success rate on DSR task: Gemini 3.1 Pro and GPT-5.5: 100%, Opus 4.7 and Sonnet 4.6: 90%, Haiku 4.5: 70%, Qwen 7B: 0%
- Median iterations for DMQ: Opus 4.8 and Sonnet 4.6: 1 iteration vs Haiku 4.5: 2 iterations
- Top-k ranking overlap and severity score error: all top-performing LLMs showed small absolute errors compared to ground truth KPIs (exact numerical deltas in Table V)
- Median run time for DSR: GPT-5.5 05:43 mins vs Haiku 4.5 11:47 mins
Limitations
- Benchmark uses simplified and publicly modifiable dynamic models, which may not fully capture proprietary industrial model complexity or confidentiality constraints.
- The agentic evaluation currently excludes human-in-the-loop approval steps common in real workflows, limiting assessment of explainability and trust.
- Simulation environments and action spaces are limited in scope (e.g., only 4 gains modifiable in DMQ) to maintain tractability but may not generalize to fully unconstrained scenarios.
- Agent evaluation is probabilistic with stochastic LLM output and tool-call timing, introducing run-to-run variability despite fixed task definitions.
- Tool execution failures (not reasoning errors) contributed to evaluation failures for some agents, indicating robustness challenges in integration pipelines.
- No adversarial attack or robustness evaluation of the agents or underlying simulation tools was conducted.
Open questions / follow-ons
- How can human-in-the-loop approval processes and uncertainty explanations be integrated into agentic benchmarking for greater alignment with engineering practice?
- To what extent can agent robustness and fault tolerance be improved against simulator or tool execution failures observed in some runs?
- What extensions are needed to incorporate electromagnetic transient (EMT) models, protection relay fidelity, and PMU-data replay in dynamic benchmarking?
- How might deterministic planning, ensemble agents, or retrieval-augmented generation improve stochastic reproducibility and task efficiency beyond single LLM outputs?
Why it matters for bot defense
Although PowerAgentBench-Dyn is focused on power system dynamic studies, its core design principles around agentic evaluation—multi-round interaction with constrained action spaces, tool grounding, outcome-oriented metrics, and constraint compliance—offer useful parallels for bot-defense and CAPTCHA systems that require adaptive, multi-step reasoning under operational constraints. Evaluating AI agents on their ability to interact with external tools and comply with engineering guardrails while producing auditable decisions resonates with security-sensitive workflows where bot behavior must be assessed beyond simple classification accuracy. Additionally, the benchmark’s use of probabilistic agent reproducibility metrics suggests a framework to handle stochasticity inherent in modern AI systems employed in security contexts. Practitioners in CAPTCHA design could consider adopting similar iterative, tool-augmented agent simulations to validate bot defenses or automating adversarial detection pipelines that require multi-turn interaction and constraint adherence.
Cite
@article{arxiv2606_20401,
title={ PowerAgentBench-Dyn: A Benchmark for Agentic AI in Power System Dynamic Studies },
author={ Qian Zhang and Andrea Pomarico and Costas Mylonas and Magda Foti and Alberto Berizzi and Le Xie },
journal={arXiv preprint arXiv:2606.20401},
year={ 2026 },
url={https://arxiv.org/abs/2606.20401}
}