NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems

Source: arXiv:2606.27243 · Published 2026-06-25 · By Shaohua Liu, Liang Fang, Yilong Sun, Shudong Huang, Qingsong Luo, Xiaoyang Chen et al.

TL;DR

This paper addresses the challenge of automating architecture evolution in large-scale industrial recommender systems, focusing on producing valid, high-quality model upgrades under strict production constraints. Existing automation methods, such as AutoML hyper-parameter tuning or generic coding agents, do not sufficiently capture cross-module, topology-level modifications and often generate runnable but semantically invalid or ineffective candidates that cause silent performance failures. NOVA introduces a novel agent harness that integrates an "architecture gradient" — a structured, non-differentiable update signal aggregating prior modifications, verification diagnostics, metric feedback, and trajectory history — to guide iterative architecture modifications more effectively.

The system implements a multi-stage verification cascade that checks candidates for architecture-semantic correctness, local executability, offline effectiveness, and eventually online business impact, filtering out invalid or negative proposals early and propagating failure patterns as forbidden directions to improve search efficiency. NOVA further employs a level-aware task control (L1–L4) to match automation with task complexity and risk, deploying a human-in-the-loop "Copilot" mode for high-risk innovation tasks. Deployed in a large-scale advertising recommender serving billion-scale user traffic, NOVA achieves significantly higher effective pass rates on architecture modification tasks (up to 60%) compared to human expert loops and coding-agent baselines, reduces silent failures, and speeds literature-to-production cycle time by 13×. Online A/B tests demonstrate consistent gains in key business metrics such as GMV (+1.25% to +2.02%) and reduced prediction bias, validating the approach's practical impact.

Key findings

• NOVA achieves the highest effective pass rates (EPR) on L2 ScaleUp and L3 Literature-to-Production tasks: 54.5% and 60.0%, surpassing human experts and coding-agent baselines.
• NOVA reduces silent failure rates compared to coding-agent baselines by explicitly filtering runnable but semantically invalid architectures before training.
• NOVA shortens one literature-to-production architecture transfer cycle by over 13x in human-attended time compared to manual expert iterations.
• Online A/B tests of the selected L3 architecture candidate show improvements in GMV of +1.25%, +1.70%, and +2.02% across three pCVR objectives.
• The same online candidates reduce prediction bias by 58.8%, 66.7%, and 37.3% on the corresponding pCVR metrics.
• Local Pass Rate (LPR) for human experts is high but with lower effective pass rates due to slower iteration; AutoML and coding agents have lower LPR due to ignoring architectural coupling constraints.
• Semantic verification gates ahead of expensive training reduce repeated invalid candidates by recording failure patterns as forbidden directions in trajectory memory.
• NOVA’s architecture gradient combines prior modification vectors, verification outcomes, offline metric deltas, and trajectory memory to guide search directions analogous to SGD.

Threat model

The adversary is conceptualized as the complexity and risk inherent in automating architecture modifications, where invalid or semantically incorrect architectures may degrade model or business performance silently. The system assumes no malicious adversary attacking infrastructure but focuses on inadvertent generation of runnable yet architecturally invalid or suboptimal candidates. NOVA’s multi-stage verification and forbidden direction feedback aim to thwart this ‘silent failure’ threat class.

Methodology — deep read

The paper tackles architecture evolution as an iterative discrete optimization over large, constrained feasible spaces representing model graphs, structural hyper-parameters, and feature configurations (A_t = (G_t, ϕ_t, F_t)). The goal is to discover architecture modifications that improve offline AUC and eventually online business metrics under strict production constraints such as shape/dtype compatibility, latency, and training/inference stability.

(1) Threat Model & Assumptions: The adversary is effectively complexity and risk in automated architecture evolution. The system assumes access to production codebases, historical architecture priors from papers and a knowledge base, and controlled offline/online evaluation pipelines. The model constraints and correctness form hard safety boundaries. The adversary cannot violate production infrastructure integrity but can propose invalid or silent-failure-prone architectures.

(2) Data: The evaluation uses a proprietary industrial-scale production dataset of user-item interactions over one month with billions of interactions and 1000+ features including sequences and nonsequences. Offline training, verification, and online A/B tests occur on live traffic data. Split details are as per industry standard practice but not formally documented in the paper.

(3) Architecture/Algorithm: NOVA maintains a dynamic architecture state and a modification search space. It defines an "architecture gradient" g_t combining: (a) previous modification vector e_{t-1}; (b) verification diagnostics V_t from semantic and executability gates; (c) change in offline AUC metric ΔJ_t; and (d) historical trajectory memory H_t recording successes/failures. This gradient guides selection of the next candidate modification from the feasible set E. Verification consists of a cascade of semantic checks (shape/dtype/masking/fusion consistency), local execution tests, offline training with AUC evaluation, and selective online validation.

(4) Training Regime: Each candidate architecture is trained from scratch on the production dataset with fixed protocols. The search runs with max N=10 iterations per task, generating K candidates each iteration. The same offline budget applies uniformly to NOVA and baselines. NOVA integrates human confirmation via Copilot mode for high-complexity or uncovered skill tasks (L3/L4 levels).

(5) Evaluation Protocol: Metrics include Local Pass Rate (ratio passing local tests), Silent Failure Rate (runnable but negative candidates), Effective Pass Rate (LPR multiplied by success ratio), offline AUC delta, and online business impact (GMV and bias). The protocol compares NOVA against baselines such as human expert logs, generic LLM coding agents (ReActAgent, OpenHands), and AutoML (Optuna-TPE). Statistical significance details are not explicitly stated. Ablations test components like feedback inclusion, multi-candidate generation, and verification. Online A/B tests validate selected offline candidates in production.

(6) Reproducibility: The dataset is proprietary and not public. The code, model weights, and exact implementation details are not publicly released. The design emphasizes industrial deployment context. Concrete example: In the L3 Literature-to-Production task, NOVA uses architecture gradient feedback from verification failures and AUC metric changes to reject invalid TokenMixer-Large transplant candidates early, preventing wasted training; only high-quality candidates with semantic correctness and offline AUC improvements proceed to online evaluation, achieving 60% EPR and delivering positive GMV lifts in live traffic.

Overall, the methodology tightly integrates structured verification and metric-driven feedback into an SGD-inspired discrete architecture search loop, augmented with multi-level automation and human-in-the-loop pathways, to successfully evolve complex industrial recommender structures.

Technical innovations

• Introduction of the architecture gradient, a structured, non-differentiable update signal aggregating past modification directions, verification diagnostics, offline metric deltas, and trajectory memory to guide architecture evolution.
• A silent-failure-aware multi-stage verification cascade that performs architecture-semantic validation before expensive offline training and uses failure patterns as forbidden directions to steer the search away from invalid candidates.
• A level-aware task control framework (L1–L4) that adaptively routes modifications through automatic pipelines (AutoRun) or human-in-the-loop Copilot assistance based on task complexity and risk.
• Integration of architecture verification diagnostics directly into the search feedback loop, making verification an active component rather than passive filtering, thus reducing silent failure rates and improving effective pass rates.

Datasets

• Proprietary industrial advertising recommendation dataset — billion-scale user-item interactions over one month — collected from production traffic at Tencent; not publicly available.

Baselines vs proposed

• Human Expert Loop: effective pass rate (EPR) < NOVA’s 54.5% on L2 ScaleUp and 60.0% on L3 Literature-to-Production tasks.
• ReActAgent-only coding agent: lower local pass rate and effective pass rate than NOVA (exact numbers not specified, but relative decrease shown in Fig 5).
• OpenHands coding agent: optimizes for runnable code but suffers higher silent failure rates compared to NOVA.
• Optuna-TPE (AutoML): applicable only to L2 ScaleUp, achieves lower EPR due to ignoring architecture coupling constraints, e.g., producing invalid divisible token dimension/count pairs.
• Online Selected L3 Candidate: GMV improved by +1.25%, +1.70%, +2.02% on three pCVR objectives; Bias reduced by 58.8%, 66.7%, and 37.3% respectively.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27243.

Fig 1: Overview of the NOVA level-aware architecture-gradient workflow. The Main Agent fixes the task level and execution

Fig 2: Silent-failure-aware verification cascade. 𝐾can-

Fig 3: Future evolution roadmap of NOVA across full-

Limitations

• Dataset and code are proprietary, limiting third-party reproducibility and benchmarking.
• Online evaluation is reported only for a single selected candidate rather than full search trajectories or multiple runs.
• Verification relies on fixed skill specifications, which may not entirely cover future unseen architectural innovations, requiring ongoing manual updates.
• The architecture gradient is heuristic and inspired by SGD analogy but lacks formal mathematical guarantees or convergence proofs.
• Human-in-the-loop tasks (L3/L4) require manual confirmation, introducing potential delays and subjective biases.
• No detailed adversarial evaluation against deliberate invalid candidate injection or robustness testing under noisy feedback is reported.

Open questions / follow-ons

• How to formally characterize and optimize the architecture gradient for improved theoretical convergence and robustness?
• Can NOVA’s skill specifications and verification gates be generalized or automatically learned for other recommendation backbones or domains?
• What are the trade-offs in human-in-the-loop involvement for increasingly complex L3/L4 tasks, and can more autonomy be safely gained?
• How effective is NOVA under concept drift or distributional shifts in production data over longer time horizons?

Why it matters for bot defense

While this paper focuses on automated evolution of recommender system architectures rather than bot defense or CAPTCHAs directly, its core insights into preventing silent failures via multi-level semantic verification and integrating feedback into iterative search have clear parallels for CAPTCHA system adaptation. Bot-defense teams could apply analogous verification cascades to candidate bot-detection model modifications, blocking seemingly valid but functionally unsound changes before deployment. Furthermore, NOVA’s architecture gradient concept suggests a structured way to aggregate prior attack-detection failures and online feedback into automated model tuning.

The level-aware human-in-the-loop control might also inform risk management in CAPTCHA system updates, routing high-risk or novel defense innovations for manual review. Overall, NOVA exemplifies how combining domain-aware constraints, feedback-driven exploration, and verification can yield robust, scalable model evolution—the same principles are critical for safely improving bot detection or CAPTCHA challenge-response architectures at scale.

Cite

@article{arxiv2606_27243,
  title={ NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems },
  author={ Shaohua Liu and Liang Fang and Yilong Sun and Shudong Huang and Qingsong Luo and Xiaoyang Chen and Dongqiang Liu and Chuangang Ma and Zhenzhen Chai and Henghuan Wang and Shijie Quan and Changyuan Cui and Zhangbin Zhu and Peng Chen and Wei Xu and Lei Xiao and Haijie Gu and Jie Jiang },
  journal={arXiv preprint arXiv:2606.27243},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27243}
}

Read the full paper

• arXiv:2606.27243 (abstract)
• arXiv:2606.27243 (PDF)