GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation

Source: arXiv:2607.11503 · Published 2026-07-13 · By Jiale Zhang, Juntao Hu, Zhijian Ou

TL;DR

The paper addresses the challenge of long-form article generation, a complex task for large language models (LLMs) due to the need to handle long context, detailed instructions, and generate coherent, well-structured long outputs. Existing multi-agent pipelines like STORM improve content coverage but suffer from tightly coupled roles and heuristics embedded in prompts, making iterative improvement and modular reuse difficult. To solve this, the authors propose GEIS, a Generation–Evaluation–Improvement loop framework that decomposes long-form article creation into explicit, named, and declarative agent "skills." These skills separately handle writing, evidence retrieval via browser automation, image and diagram generation, pairwise PDF-aware evaluation, and automated skill improvement through patching writing rules based on evaluation feedback.

Implemented in Tasi Harness, GEIS centers on a writing skill structured into six stages: Request, Plan, Draft, Audit, Refine, and Deliver. The evaluation skill produces symmetric, pairwise quality reports comparing articles on structure, content, visual quality, and PDF delivery. The improvement skill converts recurring evaluation findings into permanent patches to update the writing skill’s rules. Evaluations on 20 Wikipedia Featured Article topics show GEIS improves the Tasi Harness default writer by 8 points on a 100-point quality rubric and outperforms STORM on structural and content dimensions. A closed-loop 20-topic experiment applying writing skill patches further improves average quality from 82.90 to 86.95. These results demonstrate that modular skill decomposition with explicit evaluation-guided improvement can significantly boost long-form generation quality beyond fixed pipelines or undifferentiated prompting.

Key findings

• GEIS raises average PDF rubric total score from 74.1 (Tasi Harness default) to 82.1, an 8.0 point gain over 20 Wikipedia topics (Table 3).
• Structural quality improves +2.6 points, content quality +3.2 points, visual quality +2.2 points; PDF delivery quality is unchanged.
• GEIS outperforms STORM by 5.6 points in structural quality and 2.2 points in content quality on the same 20-topic set (Table 4).
• A 20-topic skill improvement experiment applying evaluation-driven patches raises average quality from 82.90 to 86.95 (+4.05 points), improving 17 out of 20 topics (Table 5 and 6).
• Content quality gains (+3.85) dominate the improvement, while structural quality increases only slightly (+0.20).
• The six-stage article-writer process creates explicit quality gates: Request, Plan, Draft, Audit, Refine, Deliver (Fig 1).
• Pairwise symmetric PDF-aware evaluation reports identify authoring issues and enable targeted skill patches rather than simple prompt tuning.
• Some topics declined slightly post-patching, indicating uniform rules need topic-sensitive adaptation.

Threat model

n/a - The paper addresses improving LLM long-form generation quality in controlled conditions rather than adversarial or security threats.

Methodology — deep read

1. Threat Model & Assumptions: The paper assumes a professional content generation scenario where the adversary is not a malicious attacker but the challenge is systematically producing high-quality, well-structured long-form articles. The methods focus on iterative improvement through modular skills rather than defending against adversarial attacks.

2. Data: The evaluation is conducted on 20 English Wikipedia Featured Articles spanning diverse topics including history, AI, software engineering, and biography. These community-reviewed articles serve as high-quality references but are not treated as gold answers. The dataset is used as topic prompts for generation, baseline quality references, and comparative evaluation.

3. Architecture/Algorithm: GEIS is implemented as a set of named, declarative agent skills deployed in the Tasi Harness runtime. The core article-writer skill structures generation into six distinct stages with explicit responsibilities: Request (task normalization), Plan (section outline with visual anchors), Draft (full prose generation), Audit (quality checks on structure, factuality, clarity), Refine (revision of weak parts), and Deliver (final output emission). Secondary skills handle browser-based web evidence retrieval (tasi-browser-automation), image and diagram insertion (architecture-diagram), pairwise symmetric PDF-aware evaluation (pdf-comparison-evaluation), and automatic writing skill patching (article-writer-improving). Evaluation outputs structured reports with rubric scores and actionable findings.

4. Training Regime: No novel model training is performed; the system uses GPT-5.4 for generation and Qwen 3.5 Plus as an independent evaluator. Improvements arise from rule-based patching of skill documents (SKILL.md) which guide LLM prompting and generation workflow rather than changing model weights. This allows lightweight iterative skill improvement without retraining.

5. Evaluation Protocol: Quality is measured by a weighted 100-point rubric across four dimensions: structural quality (35), content quality (35), visual quality (15), and PDF delivery quality (15). Pairwise evaluation compares two peer documents symmetrically, avoiding gold standard biases. Experiments include: (i) comparing GEIS with default Tasi Harness writer, (ii) comparing GEIS with STORM pipeline on structural and content dimensions, and (iii) closed-loop iterative skill improvement on 20 topics. Evaluation uses independent LLM judge to reduce bias.

6. Reproducibility: The system is implemented in the open Tasi Harness platform. Full skill markdown assets and code are planned to be open sourced upon paper acceptance. The datasets are publicly identifiable Wikipedia articles, but exact prompts and skill patches are not fully detailed here. Models used are commercial LLMs (GPT-5.4, Qwen 3.5 Plus). The experiments are deterministic workflows rather than neural training runs, facilitating reproducibility.

Concrete example end-to-end: For a Wikipedia featured article topic, the article-writer skill first normalizes the task (Request), creates a multi-level section outline with placeholders for images and diagrams (Plan), generates a full draft with integrated prose (Draft), audits the draft for structure and factual weaknesses (Audit), refines sections flagged as weak (Refine), and finally delivers the polished article text (Deliver). The generated PDF is compared pairwise against alternative outputs (e.g., baseline or reference) producing a detailed evaluation report. The article-writer-improving skill inspects recurrent findings from multiple evaluation runs, produces rule-based patches to the writing process (e.g., enforce inline citations, deeper sections), then the patched writing skill regenerates articles with improved scores.

Technical innovations

• Introducing a modular agent skill framework that separates writing, retrieval, image/diagram generation, evaluation, and improvement into auditable, declarative units.
• Structuring long-form article generation as a six-stage semantic writing pipeline (Request, Plan, Draft, Audit, Refine, Deliver) providing explicit intermediate quality gates.
• Designing a symmetric pairwise PDF-aware evaluation skill that produces detailed diagnostic reports rather than absolute scores, enabling targeted skill patching.
• The article-writer-improving skill translates structured evaluation feedback into permanent writing rule patches in skill documents, enabling lightweight process improvement without model retraining.

Datasets

• Wikipedia Featured Articles Set — 20 topics — public Wikipedia articles used as prompts, references, and quality baselines

Baselines vs proposed

• Tasi Harness default writer: total quality score = 74.1 vs GEIS article-writer = 82.1 (+8.0)
• STORM pipeline: structural quality = 22.6, content quality = 28.4 vs GEIS structural = 28.2 (+5.6), content = 30.6 (+2.2)
• Pre-improvement GEIS: total score = 82.90 vs post-improvement GEIS = 86.95 (+4.05)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11503.

Fig 1: The six-stage article-writer process. Request normalizes the user task, Plan

Fig 2: Closed-loop optimization. The writing skill generates an article, the pairwise

Limitations

• Dependency on LLM-as-a-judge evaluation introduces variance and scaling drift despite rubric design and arithmetic checks; expert human review is absent.
• Evaluation references are high-quality examples, not absolute gold standards, limiting interpretability of absolute scores.
• The uniform rule-based skill patching may harm topics needing specialized structure or depth, shown by regression on 3 out of 20 topics.
• No adversarial robustness assessment or stress testing for pathological inputs was conducted.
• PDF delivery quality did not improve, implying improvements are mostly on writing and visual anchors, not export tooling.
• Dataset diversity is moderate; 20 Wikipedia topics cover various domains but may not represent all long-form document genres.

Open questions / follow-ons

• How can topic-sensitive or conditional skill patching be automated to avoid regressions seen with uniform rules?
• Can reinforcement learning or other model-centric skill evolution methods complement the rule-based patching approach?
• How can human expert feedback better integrate into the evaluation and improvement loop for stronger factuality and stylistic control?
• Could the skill framework extend to incorporate more structured data generation such as tables or domain-specific citation formats?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on attack generation or content authenticity, GEIS demonstrates an effective methodology to control and audit LLM long-form text generation through modular skill decomposition and evaluation-guided iterative improvement. The explicit staging and skill-based architecture enable better traceability and quality gates, which could inspire stronger defenses or detection by identifying modular failures or anomalous patterns at different pipeline stages. Additionally, the pairwise evaluation strategy highlights the benefit of comparative, rubric-based quality assessment over simple absolute scoring, which aligns with assessing bot-generated content quality or originality. Although not directly addressing bot detection, the skill loop concept may be adapted to develop more human-like, but auditable, text generation processes or benchmarks for synthetic content assessment.

Cite

@article{arxiv2607_11503,
  title={ GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation },
  author={ Jiale Zhang and Juntao Hu and Zhijian Ou },
  journal={arXiv preprint arXiv:2607.11503},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11503}
}

Read the full paper

• arXiv:2607.11503 (abstract)
• arXiv:2607.11503 (PDF)