Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Source: arXiv:2607.08758 · Published 2026-07-09 · By Yifan Zhou, Qihao Yang, Yan Li, Donggang Li, Xiru Hu, Hokin Deng et al.

TL;DR

This paper addresses the challenge of evaluating whether AI systems can accurately reason about and generate scientific ideas grounded in their lineage, i.e. the inheritance, modification, and recombination of ideas over successive research papers. The authors introduce IdeaGene-Bench (IG-Bench), a novel benchmark built around a structured framework representing scientific work as sets of minimal, typed "Idea Genome" objects linked by "GenomeDiff" alignments that encode scientific inheritance dynamics such as mutation and hybridization. IG-Bench comprises 1,961 gold-standard lineage traces, 1,085 curated Idea Genomes, and 920 GenomeDiff records across 10 scientific domains. It supports two evaluation modes: IG-Exam, which tests lineage reasoning by decomposing papers into genomes and verifying inheritance chains, and IG-Arena, which evaluates generation of lineage-coherent new scientific proposals scored by a lineage-conditioned Population-Evolution Score (PES) capturing heredity, meaningful variation, and future selection potential.

The experiments with 14 LLM-based scientific reasoning and generation systems reveal major bottlenecks. On lineage reasoning tasks (IG-Exam), the best model attains only 27.3% exact accuracy, with failures predominantly due to inconsistent compositional reasoning about parent linkage, driver mechanisms, and idea fate. Tool-based retrieval scaffolds improve multi-genome tracing but barely help deeper evolutionary reasoning or lineage verification. In generation (IG-Arena), lineage context helps separate systems that leverage formal lineage evidence from those relying on more surface textual cues, but overall generation coherency remains limited. Generated ideas often appear plausible but fail key lineage-grounded consistency checks, highlighting the critical gap between fluency and genuine scientific inheritance understanding and creativity. This paper redefines automated scientific evaluation beyond topical relevance or novelty, toward precise lineage-grounded idea tracing, verification, and grounded ideation.

Key findings

• IG-Bench contains 1,961 gold lineage traces, 1,085 curated Idea Genome objects, and 920 GenomeDiff records across 10 scientific domains.
• IG-Exam contains 42 task types and 1,029 instances testing genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification with exact-match scoring.
• Strongest system (GPT-5.5 + Claude Code) reaches only 27.3% exact accuracy overall on IG-Exam, showing a compositional bottleneck.
• Performance declines from Genome Abstraction (best 34.4%) to Lineage Verification (best 17.4%), reflecting increasing compositional reasoning complexity.
• CLI tool scaffolds improve T2 Inheritance Tracing accuracy substantially (e.g., GPT-5.5 from 25.7% to 37.9%) but yield little or negative gain on T3/T4 tasks.
• IG-Arena Population-Evolution Score (PES) shows lineage context improves generation quality unevenly (+4.4 median gain from Question-only to Lineage context), mainly by improving Heredity dimension.
• Variation and Selection scores remain relatively flat across settings; Heredity drives the PES improvements from lineage grounding.
• Fluency-preferred proposals by ELO ranking may be lineage-incoherent, underscoring PES as a necessary lineage-grounded generation metric.

Threat model

The threat model is an AI system attempting to emulate scientific reasoning and idea generation, where errors arise from inability to correctly identify and preserve idea inheritance structures between papers. The system must not mistakenly claim lineage connections where they do not exist or fail to verify coherent mechanism inheritance. The adversary is not malicious but suffers from compositional reasoning limitations, retrieval errors, or hallucinations. There is no consideration of external attackers, data poisoning, or deception.

Methodology — deep read

1. Threat Model & Assumptions: The paper assumes the adversary is an AI system tasked with scientific idea understanding and generation—specifically lineage competence. The model should accurately identify, verify, and extend Idea Genome structures underlying prior work, distinguishing true scientific inheritance from topical co-location without lineage. There is no adversarial attack or manipulation assumed.

2. Data: IG-Bench is constructed by expert annotation across 10 scientific domains (e.g., NLP, computer vision, biology, physics). It includes 1,961 golden lineage traces of paper sequences (3-7 papers each), 1,085 curated Idea Genomes (minimal typed idea objects like mechanism, limitation, delta), and 920 pairwise GenomeDiff alignments categorizing evolutionary transitions. Seeds come from known landmark papers; expanded via citations, semantic search, and domain expertise. Annotations are audited by 50 trained graduate students with 84.7% inter-rater agreement on dynamics labels.

3. Architecture/Algorithm: The core IdeaGene framework abstracts papers into sets of Idea Genome objects G(p) = {gi} where each genome gi has a role type (mechanism, limitation, etc.), content description, and textual evidence pointer. GenomeDiff aligns genomes between predecessor and successor papers, marking each source genome as inherited, mutated, lost, or untreated, and targets as novel or externally imported. Six operational evolutionary dynamics (mutation, adaptive radiation, hybridization, speciation, niche competition, isolation) classify these transitions, enabling fine-grained lineage reasoning.

4. Training Regime: No training per se; evaluation is done on 14 existing LLM-based systems (including GPT-5.5, Claude Opus 4.7, and derivatives) applied to both reasoning (IG-Exam) and generation (IG-Arena). CLI tool-based harnesses wrap these LLMs with information retrieval or generation scaffolds.

5. Evaluation Protocol: IG-Exam: closed-form tasks with 42 types over 1,029 instances, exact-match scoring on lineage subfields (parent identity, driver mechanism, lineage dynamics, genome fate), testing genome abstraction through complex lineage verification. IG-Arena: open-ended generation scored by a Population-Evolution Score (PES) that measures heredity (correct inheritance), variation (meaningful novelty), and selection (future research potential), averaged over judge panel ratings under controlled conditions (Question-only, Library without lineage structure, and full Lineage context). Preference metric ELO from pairwise battles complements PES.

6. Reproducibility: The paper offers a public project page with benchmark data and an evaluation framework. Though some curated data derives from expert annotation (non-public domain papers), the benchmark and evaluation code are released for community use. LLM weights are closed-source.

Example: For an IG-Exam task instance, the system receives a set of Idea Genome objects extracted from papers and is asked to trace which genome components persist, mutate, or are lost when moving from paper A to paper B, classify the evolutionary dynamics (e.g., mutation or hybridization), and verify if a claimed lineage proposition is consistent with given genomic evidence. Exact accuracy demands the system get all subfields (parent, driver, fate, rationale) correctly. Error analysis shows models sometimes identify parents correctly but err in driver assignment or genome fate, revealing challenges in compositional consistency.

Technical innovations

• IdeaGene framework: a structured representation of scientific papers as sets of typed, minimal, evidence-grounded Idea Genome objects enabling auditable lineage comparison.
• GenomeDiff operation: alignment and classification of genome objects across predecessor/successor papers into inheritance categories (Inherited, Mutated, Lost, External, Novel) with driver and rationale labels.
• Operational evolutionary dynamics taxonomy (mutation, adaptive radiation, hybridization, speciation, niche competition, isolation) capturing nuanced scientific lineage relations beyond citation or topical overlap.
• Lineage-grounded Population-Evolution Score (PES) metric decomposing generated idea quality into heredity, variation, and selection for lineage-conditioned evaluation in open-ended generation.

Datasets

• IdeaGene-Bench (IG-Bench) — 1,961 lineage traces, 1,085 curated Idea Genome objects, 920 GenomeDiff records — expert curated across 10 scientific domains

Baselines vs proposed

• GPT-5.5 IG-Exam exact accuracy = 23.1% total vs GPT-5.5 + Claude Code harness = 27.3%
• T1 Genome Abstraction: GPT-5.5 27.5% vs GPT-5.5 + Codex 31.8%
• T2 Inheritance Tracing: GPT-5.5 25.7% vs GPT-5.5 + Claude Code 37.9%
• T4 Lineage Verification: GPT-5.5 16.0% vs GPT-5.5 + Claude Code 12.7% (no improvement)
• IG-Arena PES metric: GPT-5.5 scores 86.5 in Lineage setting vs 85.2 in Question-only; median system gain +4.4 from Question to Lineage
• ELO ranking vs PES correlation Spearman ρ=0.82; PES better captures lineage coherence over fluency-based preferences

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08758.

Fig 1: From paper-centric search to genome-centric lineage. A paper-centric view leaves a model with

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 2).

Fig 2: IG-Bench evaluation design. IG-Bench converts input papers into a lineage substrate of audited Idea

Fig 3: IG-Arena PES analysis. (a) Question–Library–Lineage trajectories show that structured lineage context

Fig 5: Domain-level PES snapshot. Lineage-setting PES across 10 domains for the evaluated systems. Strong

Fig 6: PES decomposition and information-setting breakdown. (a) Heredity, Variation, and Selection

Limitations

• Benchmark construction relies on expert annotation and domain knowledge with some subjectivity despite auditing; potential label noise or bias possible.
• Evaluation focuses on 10 scientific domains, limiting generalization to others; some domains less detailed or smaller sample size.
• Generation evaluation restricts proposals to pre-defined lineage populations, may not cover extremely novel ideas outside those.
• Exact-match scoring penalizes partially correct lineage reasoning, which might underrepresent incremental understanding.
• No explicit adversarial robustness or manipulation evaluation against malicious lineage claims or fake ideas.
• Dependence on closed-source LLMs and tool scaffolds limits reproducibility and interpretability of system failures.

Open questions / follow-ons

• How to design compositional verification modules to improve consistent lineage reasoning across roles, drivers, and genome fates?
• Can automated genome extraction and diff alignment be fully automated without expert auditing to scale the benchmark?
• How can lineage-grounded generation be integrated into end-to-end autonomous research workflows for iterative hypothesis synthesis?
• What neural architectures or training objectives best capture operational evolutionary dynamics in idea inheritance?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights the nuanced challenge of assessing and generating content that respects complex inheritance structures rather than superficial topical relevance or fluency. While CAPTCHAs rely on distinguishing genuine human reasoning or behavior patterns from automated responses, the IdeaGene framework offers a blueprint for modeling compositional reasoning about lineage analogously. In bot-defense, understanding lineage grounding could inspire advanced puzzles or verification challenges demanding deep structural consistency rather than pattern matching. Furthermore, evaluation metrics like PES emphasize multi-dimensional assessment beyond simple correctness, which could inform richer CAPTCHA scoring strategies that verify coherence over sequences of challenge steps. Although the domain focus is scientific ideas, the core concepts of tracing inheritance and verifying consistent composition may translate to areas where bots attempt to mimic human-generated knowledge or workflows.

Cite

@article{arxiv2607_08758,
  title={ Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation },
  author={ Yifan Zhou and Qihao Yang and Yan Li and Donggang Li and Xiru Hu and Hokin Deng and Ziyang Gong and Xuanyi Zhou and Huacan Wang and Xiangchao Yan and Wanghan Xu and Wenlong Zhang and Shaofeng Zhang and Yue Zhou and Yifan Yang and Zhihang Zhong and Xue Yang },
  journal={arXiv preprint arXiv:2607.08758},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08758}
}

Read the full paper

• arXiv:2607.08758 (abstract)
• arXiv:2607.08758 (PDF)