Agentic Hardware Design as Repository-Level Code Evolution

Source: arXiv:2606.28279 · Published 2026-06-26 · By Cunxi Yu, Chenhui Deng, Nathaniel Pinckney, Brucek Khailany

TL;DR

This paper introduces HORIZON, a novel framework for agentic hardware design that treats hardware design problems as evolving code repositories managed via git. Unlike prior work focusing on single-shot RTL generation or iterative repair limited to isolated modules, HORIZON hosts the entire hardware design task—including RTL code, testbenches, verification artifacts, and domain knowledge—as a self-contained git worktree, with an executable evaluator and acceptance predicate gating commits. The agent loop continuously generates, edits, evaluates, accepts or rejects candidate solutions, leveraging git history for state tracing and replayability. This repository-native approach extends concepts previously applied to EDA software self-evolution to directly evolve hardware designs and verification, providing a hands-free, fully agent-driven design evolution process. The authors evaluate HORIZON on multiple RTL benchmarks including ChipBench, RTLLM, Verilog-Eval, and nine CVDP categories covering generation, modification, reuse, testbench generation, assertion generation, and debugging. HORIZON achieves 100% pass rates across all suites through iterative agentic evolution, markedly improving over initial first-iteration pass rates and addressing even challenging verification tasks that single-turn models struggle with. The paper thoroughly analyzes the convergence behavior, token consumption, and test coverage dynamics, while acknowledging that agentic hardware design is far from solved in practical chip engineering contexts. Limitations and open problems in convergence efficiency, verification quality, and real project constraints motivate future work.

Key findings

• HORIZON achieves 100% pass rates on ChipBench, RTLLM-2.0, Verilog-Eval-v2, and nine CVDP RTL benchmark categories, including complex verification tasks (Table 1).
• Initial iteration pass rates vary widely, e.g., 47.8% aggregate across legacy suites but as low as 3.2% on CVDP code completion (CID 002) and 3.8% on checker generation (CID 013), illustrating the need for iterative repair rather than one-shot generation.
• Convergence iterations range from 1 (CID 014 assertion generation) to 82 (CID 002 code completion), showing some benchmarks require extensive agent iteration to reach correctness.
• Token consumption is dominated by the hardest CVDP categories, with CID 002 alone consuming 56M tokens and accounting for 26.7% of totals (Table 2).
• About 91% of tokens consumed are cached inputs across iterations due to persistent session reuse, lowering actual API cost.
• Testbench stimulus generation (CID 012) improves average coverage from 86.5% to 97.9% as pass rates reach 100%, demonstrating convergence halts at pass gate rather than maximal coverage (Table 3, Figure 4).
• Repository commit history traces accepted and rejected attempts, enabling reproducible, replayable agent evolution traces with full evaluator evidence.
• The agentic loop enables many low first-pass-rate tasks to steadily climb to 100% correctness rather than plateauing, e.g., CID 013 checker generation rises steadily to perfect pass rate by iteration 19.

Threat model

n/a — The paper does not focus on adversarial threats in the classical security sense, but rather on the challenge of autonomously evolving hardware designs to meet executable correctness criteria using iterative LLM agent actions gated by verification feedback.

Methodology — deep read

The paper tackles agent-driven iterative hardware design within a repository-level code evolution paradigm. Threat Model: The adversary model is not a traditional attacker but the challenge is to develop an agent that autonomously evolves hardware designs to satisfy strict executable correctness gates, relying on tool-based verification feedback but without perfect oracle knowledge. Data and Benchmarks: Evaluation uses multiple public RTL benchmark suites: ChipBench, RTLLM-2.0, Verilog-Eval-v2, and nine categories from the CVDP dataset totaling 783 human-authored problems spanning RTL generation, modification, testbench/stimulus generation, assertion/checker generation, and debugging. The suite includes open-source and commercial EDA backends as evaluators, which compile, simulate, and verify designs. Architecture: HORIZON is a framework rather than a specific model; it uses a fixed LLM agent backbone (GPT-5.3) combined with a bootstrap compiler that converts a user-supplied Markdown harness into a project pack, specifying domain knowledge, evaluator, acceptance predicate, and git/runtime policies for the task. The agent operates over an isolated git worktree corresponding to the design repo, applying edits as patches. The agent loop is free-form and history-dependent, modeled as a semi-Markov decision process over repository snapshots. Each iteration involves candidate artifact generation, tool calls (e.g., compilation, simulation), evaluator evidence gathering, and predicate-based acceptance with commits or reject logs. Git semantics underpin state management, traceability, version diff inspection, and replayability, forming a detailed experience buffer naturally. Training Regime: The GPT-5.3 backbone is fixed; no RL training or model updates occur within the experiments. The architecture relies on persistent session token caching to amortize prompt size and reduce marginal token cost during long iterative campaigns. Iterations proceed automatically without human intervention. Evaluation Protocol: Results track pass rates on benchmarks by iteration number, along with token consumption totals, pass predicate satisfaction, and auxiliary metrics like test coverage for generation tasks where available. Iteration-0 results evaluate the state after one full agent step, reflecting agentic evolution rather than pure first-shot modeling. Benchmarks cover open and commercial EDA flows, and all runs occur on a 32-core AMD EPYC processor with ample RAM. Single-agent manual-free runs produce final results. The repository history captures accepted commits and rejected attempts with evaluator outputs attached as git notes for independent audit. Reproducibility: The authors note that the methodology is platform and tool dependent; benchmarks and frameworks are drawn from open sources such as CVDP, RTLLM, and Verilog-Eval, but the GPT-5.3 model and some commercial EDA backends are proprietary. While examples and data are well documented, no explicit public code release or frozen weights are provided in this version. Overall, the methodology builds a novel experimental setup combining LLM-driven generation, executable correctness harnesses, and git-native version control to instantiate a fully autonomous iterative RTL design agent loop proven on diverse hardware design benchmarks.

Technical innovations

• Introducing a git-native agent framework treating hardware design tasks as evolving version-controlled repositories rather than single-shot prompts.
• Compiling a human-supplied Markdown harness into a structured project pack that embeds domain knowledge, executable evaluators, acceptance predicates, and runtime policies, enabling versatile hardware-design benchmarking.
• Framing repository states as semi-Markov process snapshots and candidate changes as variable-length episodic options, enabling detailed replayable agent experience traces integrated with native git logs and notes.
• Implementing a fully hands-free LLM agent loop that autonomously edits, evaluates, commits, or rejects candidate hardware design versions with no human intervention across diverse RTL benchmarks.
• Leveraging persistent model session token caching to drastically reduce API costs and enable multi-iteration exploration on challenging verification and synthesis tasks.

Datasets

• ChipBench — varies, public RTL benchmark
• RTLLM-2.0 — natural language to RTL mapping tasks, public
• Verilog-Eval-v2 — HDLBits-style Verilog generation problems, public
• CVDP — 783 human-authored RTL code and verification problems across 13 categories, public

Baselines vs proposed

• Iteration 0 pass rate on CVDP CID 002 code completion: 3.2% vs final pass 100% by iteration 82
• Iteration 0 pass rate on CVDP CID 013 checker generation: 3.8% vs final pass 100% by iteration 19
• Verilog-Eval iteration 0 pass rate: 86.2% vs final pass 100% by iteration 2
• RTLLM-2.0 iteration 0 pass rate: 78.0% vs final pass 100% by iteration 2
• ChipBench iteration 0 pass rate: 20.0% vs final pass 100% by iteration 5

Limitations

• Benchmarks are controlled proxies and do not represent full complexity of real chip projects, which involve incomplete specs, integration, human review, and PPA tradeoffs.
• Reward feedback is tailored to benchmark pass conditions, not to richer signals like synthesis quality-of-results or exhaustive coverage maximization.
• Agent session uses a fixed LLM backbone with no learning or policy updates during runs, limiting adaptability and efficiency improvements.
• Experiments rely partly on proprietary GPT-5.3 and commercial EDA tools, impacting reproducibility and generality.
• No explicit robustness tests against adversarial or unexpected inputs, distribution shifts, or specification changes were reported.
• The approach favors convergence completeness over efficiency; token consumption for difficult benchmarks can be very large.

Open questions / follow-ons

• How to improve token efficiency and convergence speed on the hardest hardware design tasks without sacrificing completeness?
• Can richer reward signals incorporating synthesis PPA, coverage maximization, or multi-objective tradeoffs be integrated effectively?
• What mechanisms can prevent reward hacking or overfitting to benchmark-specific acceptance predicates?
• How to extend the approach to incorporate evolving or partial specifications and human-in-the-loop workflows encountered in real chip projects?

Why it matters for bot defense

While HORIZON focuses on agentic hardware design rather than bot defense or CAPTCHA specifically, it offers insights into designing autonomous agentic systems governed by executable feedback and traceable version control. The principle of framing complex generation tasks as stateful repository evolution with executable correctness gates may inspire bot-defense architectures that manage evolving proof-of-work challenges or multi-step puzzle validation with verifiable history. Moreover, the emphasis on handling long-horizon iterative repair with rich tool feedback could translate to adaptive CAPTCHA schemes that evolve in response to adversarial attempts, backed by recorded trial histories. However, direct application would require adapting domain-specific evaluators and acceptance criteria to bot-detection tasks rather than hardware verification.

Cite

@article{arxiv2606_28279,
  title={ Agentic Hardware Design as Repository-Level Code Evolution },
  author={ Cunxi Yu and Chenhui Deng and Nathaniel Pinckney and Brucek Khailany },
  journal={arXiv preprint arXiv:2606.28279},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28279}
}

Read the full paper

• arXiv:2606.28279 (abstract)
• arXiv:2606.28279 (PDF)