An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

Source: arXiv:2607.27877 · Published 2026-07-30 · By Yanyu Ren, Yunfeng Bai, Xizheng Wang, Li Chen, Dan Li

TL;DR

This paper addresses the critical gap in evaluating multi-agent AI systems that collaboratively develop full-stack software projects from scratch. Existing benchmarks largely assess isolated functions or single-agent patching, often in synthetic or simulated environments, failing to capture real-world costs, deployment complexities, and iterative improvement. The authors introduce MSEval, a comprehensive multi-agent from-scratch evaluation benchmark grounded in 10 authentic, large-scale university capstone projects spanning diverse domains. MSEval couples a multi-agent execution engine (LegoGent) supporting 10 coordination topologies and native CI/CD deployment, with an automated rubric-driven grading system (TAgent) that probes live running artifacts for functional correctness, latency, and token use, across multiple refinement rounds. This design enables measuring speed, cost, and quality trade-offs rigorously and reproducibly under different collaboration policies.

Empirical results from 100 runs across 10 modes, 10 projects, and multiple LLMs reveal that collaboration topology has an equal or greater impact than model choice on final outcomes. Structured, pipeline-like coordination yields the fastest, highest-quality code, while heavy managerial oversight or loosely constrained open-source review degrade performance and increase resource use. The benchmark detects typical software engineering failure modes—prerequisite cascade failures, incomplete logic, and deployment integration issues—and demonstrates the importance of accountability, bounded parallelism, and iterative repair informed by granular automated feedback. MSEval thus provides a practical, realistic testbed for studying multi-agent coding coordination beyond traditional single-agent metrics.

Key findings

• Varying collaboration topology shifts performance scores by over 30 points on a 0–100 scale and can double wall-clock development time for identical tasks and models.
• The structured pipeline coordination mode converges fastest with the highest code quality among 10 tested collaboration modes.
• Heavily manager-supervised modes (PM oversight) often degrade speed and quality compared to more autonomous modes like feature squads or QA-first.
• Functional completion scores improved across refinement rounds in 82% of cases, with 95% of runs finishing round 3 above initial scores, showing effective use of TAgent feedback.
• GLM-5.2 model achieved the highest average full-grid score but at roughly 2.6x the wall-clock time and 8x the USD cost of DeepSeek v4 Pro, demonstrating a speed-cost-quality tradeoff.
• Common failure causes include incomplete logic/edge cases (32.2% of failing checks), missing features (23.8%), and prerequisite cascade failures (11%), indicating multi-agent coordination challenges.
• Deployment issues related to security and transport (8.5% of failing checks) often serve as an integration bottleneck rather than coding errors.
• QA-first and rotation modes rank highest in average performance stability, consistency, and ownership clarity across 10 projects.

Threat model

The threat model is non-adversarial, focusing on multi-agent coordination challenges arising from task division, synchronization barriers, merge conflicts, and deployment integration failures. Agents are assumed cooperative but subject to loosely bounded communication and partial ownership ambiguities. The goal is to measure realistic software engineering failures rather than security attacks.

Methodology — deep read

The authors design MSEval as a controlled multi-agent software development benchmark with the following key steps:

1. Threat Model & Assumptions: The adversary is not explicitly modeled as malicious; rather, the study focuses on coordination policies influencing multi-agent system failures such as merge conflicts, ownership ambiguity, and deployment failures. Agents are specialized LLM-based coding assistants coordinating under variable topologies; there is no adversarial manipulation assumed.

2. Data: MSEval uses 10 authentic, large-scale university capstone projects from real software engineering curricula since 2018. Domains include instant messaging, asset management, live teaching, etc. Each project is specified via hierarchical requirements documents decomposed into 6-8 modules and 30-45 weighted rubric items normalized to a 100-point scale. Projects contain ~2000+ lines of Python code after completion.

3. Architecture/Algorithm: LegoGent executes teams of 4 isolated agents running as separate processes sharing a filesystem and shell environment. It implements 10 collaboration modes controlling ownership maps, activation schedules, decision rules, and handoff artifacts. Agents communicate via periodic sync intervals (every 4 minutes) broadcasting shared states and an active mailbox for targeted messages. Completion gates validate readiness before deployment. TAgent automates grading by parsing requirements into YAML rubric, dynamically discovering UI/API/code surfaces, and running structured probes to measure functional correctness, latency, and token costs.

4. Training Regime: Although LLM model details vary (Claude Opus 4.8, GPT-5.5, DeepSeek variants, GLM-5.2), the benchmark fixes model prompt context and runs each experiment with up to 3 iterative build-refinement rounds. Precision about training epochs or hyperparameters is not applicable as models are off-the-shelf; randomness is mitigated via multiple runs.

5. Evaluation Protocol: Metrics include a hierarchical 0-100 functional completion score, wall-clock development time, USD token cost based on cloud pricing tables, and total tokens used. Scoring accounts for deployment success/failure via CI/CD pipelines integrated with GitLab and SonarQube. Baselines include different collaboration modes and multiple LLM backends. Ablations analyze refinement trajectories (R1-R3), failure taxonomy, and coordination-mode radar plots measuring ownership clarity, parallelism, speed, and consistency.

6. Reproducibility: The MSEval benchmark, LegoGent runtime, and TAgent grader are publicly released (GitHub link provided). Project specs, code, evaluation scripts, and metrics are included. Models used are commercial or open-source pretrained LLMs, with no frozen checkpoint proprietary weights shared.

End-to-End Example: For the instant messaging project, a 4-agent team runs under the Rotation mode with periodic sync and role swaps each round. Agents implement decomposed requirements (frontend, backend, QA). LegoGent coordinates code merges, triggers CI/CD deployment on GitLab, and TAgent performs multi-modal probing on the live staged app. Feedback scores and detailed failure points (e.g., missing API validation) guide iterative R2 and R3 repairs, improving functional score from initial ~76 to 84 by R3, with logged cost and time trends enabling tradeoff analysis.

Technical innovations

• MSEval is the first benchmark to treat multi-agent coding coordination topology as the primary independent variable in from-scratch full-stack software development tasks.
• LegoGent runtime supports bounded asynchronous multi-agent execution with periodic synchronization and validated artifact handoff via native CI/CD integration for realistic deployment feedback.
• TAgent graders dynamically discover UI, API, and code implementation surfaces without fixed specification schemas, enabling functional correctness evaluation on live deployed applications.
• Integration of hierarchical, weighted rubrics with multi-round actionable feedback facilitates precise repair-driven iteration rather than one-shot assessment.
• MSEval jointly measures speed, token-based cost, and quality in a unified framework, enabling quantitative tradeoff characterization under different organizational modes.

Datasets

• MSEval Projects — 10 full-stack web applications with 30–45 rubric items each — sourced from university capstone projects since 2018

Baselines vs proposed

• Claude Opus 4.8 Instant Messaging under Pipeline mode: best-round score = 91, time = 110 minutes, cost = $654 vs Open-source mode: score = 75, time = 282 minutes, cost = $190
• DeepSeek v4 Pro Feature Squad: score = 83.9, time = 83 minutes, cost = $537 vs Pipeline: score = 83.7, time = 111 minutes, cost = $491
• GLM-5.2 Pipeline mode average: score = 87.6, time = 246 minutes, cost = $57 vs QA-first: score = 80.7, time = 235 minutes, cost = $56
• GPT-5.5 Pipeline mode Instant Messaging: score approx. 91, time = 94 minutes, cost = $159 vs Swarming mode: score = 87, time = 78 minutes, cost = $255

Limitations

• Evaluation scale is limited by high computational budget; only three refinement rounds per run restricts capturing long-tail iteration patterns.
• TAgent grader, though correlated with human TA scoring (Spearman 0.87), may miss subtle defects or security issues not captured by rubric probes.
• Benchmark projects focus on web apps and university assignments, so generalization to other domains like systems programming or embedded software is untested.
• The study does not incorporate adversarial or malicious agent behavior, limiting insight into resilience against Byzantine faults or active attacks.
• Models tested are mostly closed or commercial LLMs with unstated training details, impeding analysis of training influence on multi-agent coordination.
• Some failure modes (e.g. deployment bottlenecks) may be more attributable to CI/CD infrastructure than multi-agent coding policy itself.

Open questions / follow-ons

• How can multi-agent systems dynamically adapt coordination policies (parallelism vs serialization) to optimize speed–cost–quality tradeoffs per project?
• What role can learned heuristics or meta-policies play in predicting when to stop unsuccessful development branches before resource waste accumulates?
• How well do these benchmarks and findings generalize to larger teams, more complex dependencies, or non-web software domains?
• Can continuous integration of adversarial or malicious agent strategies reveal robustness limits of coordination modes?

Why it matters for bot defense

This work highlights the crucial impact of multi-agent collaboration topologies on performance and resource use, underscoring that coordination mode can be as influential as model capability. For bot-defense and CAPTCHA practitioners deploying multi-agent workflows, this indicates that the structuring of multi-agent calls, synchronization intervals, and ownership boundaries is foundational to system efficiency and robustness. Furthermore, the blended use of deployment-grounded evaluation with automated, multi-modal probing provides a strong model for assessing production readiness rather than superficial correctness. Insights on iterative refinement driven by granular feedback are particularly relevant for costly, latency-sensitive multi-agent pipelines prevalent in anti-bot defenses.

Cite

@article{arxiv2607_27877,
  title={ An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding },
  author={ Yanyu Ren and Yunfeng Bai and Xizheng Wang and Li Chen and Dan Li },
  journal={arXiv preprint arXiv:2607.27877},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27877}
}

Read the full paper

• arXiv:2607.27877 (abstract)
• arXiv:2607.27877 (PDF)