DAGForge: Auditable Causal DAG Authoring with Biomedical Literature

Source: arXiv:2607.21859 · Published 2026-07-23 · By Yi-han Sheu, Michael R. Steigman, Yu Zhou, Bo Wang, Fan-Yu Yen, Jordan W. Smoller

TL;DR

DAGForge addresses the bottleneck in biomedical causal inference research caused by the manual and non-auditable process of constructing directed acyclic graphs (DAGs) from literature and domain knowledge. It introduces an integrated, browser-based system that automates and documents the DAG authoring pipeline, linking each causal edge to verbatim PubMed evidence and providing provenance and confidence estimates. This approach helps researchers generate reproducible, auditable DAGs grounded in biomedical studies with explicit temporal and domain constraints. Extensive evaluation against compact synthetic and real literature-derived DAGs shows DAGForge achieves high recall in edge detection while uniquely preserving traceable evidence, unlike direct large language model (LLM) baselines which produce more parsimonious but less verifiable graphs.

Key findings

• On seven synthetic biomedical DAGs (S1–S7), DAGForge achieves a median graph directed F1 of 0.895 with 0.944 pairwise skeleton F1, indicating excellent reconstruction of reference structures.
• On three literature-based DAGs without study context ('cold' setting), DAGForge attains mean graph directed F1 of 0.664 with 0.866 recall and 0.577 precision, showing high sensitivity but some surplus edges compared to parsimonious references.
• Adding study context improves literature DAG graph directed F1 to 0.714, with precision 0.625 and recall 0.905, highlighting the value of contextual information.
• 86%–87% of DAGForge’s apparent false-positive edges carry verified verbatim citations from retrieved PubMed abstracts, indicating many are valid relations missing from reference DAGs.
• Direct LLM baselines achieve higher precision but lower recall (graph directed F1 ∼0.78 on literature DAGs), and none provide verbatim, verifiable literature citations.
• DAGForge preserves full evidence provenance per edge, linking each causal judgment to exact abstract passages and citations.
• The system enforces acyclicity, temporality, and user constraints deterministically during graph assembly, reducing implausible edges.
• Evidence is sourced exclusively from frozen snapshots of PubMed abstracts; full-text evidence or specialized ontologies are not included.

Threat model

Not an adversarial security paper; the system assumes users are trusted biomedical researchers aiming to construct auditable DAGs. It does not address malicious attempts to deceive causal inference but focuses on providing provenance, citation, and reviewability to reduce user error and misplaced trust.

Methodology — deep read

DAGForge treats causal DAG construction as a staged, auditable workflow tightly integrated with biomedical literature retrieval and LLM-based reasoning.

1. Threat model: The adversary is not explicitly addressed as this is a system for expert curation rather than adversarial robustness; main assumption is that evidence grounding and user review reduce erroneous causal claims.
2. Data provenance: Users submit biomedical study variables as free text optionally enriched with UMLS ontology mappings; DAGForge constructs a reproducible PubMed abstract snapshot per run by querying concept pairs with expanded lexical blocks and causal/association relation terms. Retrieved articles are capped and filtered using a tiered strategy.
3. Architecture: The backend pipeline has these main modules: concept resolution for variable normalization; literature search to gather evidence abstracts; a retrieval-augmented LLM module (Anthropic claude-sonnet-4-6 at zero temperature) that forms structured pairwise causal judgments (relationship type, direction, confidence, citations, rationale) grounded in verbatim text snippets; and a deterministic constraint-based graph assembly unit that enforces DAG acyclicity, temporality, exogeneity, and user constraints. Graph assembly merges pairwise judgments into a coherent DAG, identifying possible latent confounders or mediated edges.
4. Training: Not a machine learning training paper; the LLM is used as a fixed inference engine with carefully designed prompting. Anthropic Claude models with fixed weights are called in a controlled, systematic manner.
5. Evaluation: Benchmarked on seven synthetic DAGs with full ground truth, and three literature-derived biomedical DAGs from published studies. Evaluation metrics include pairwise skeleton F1 (undirected connectivity), graph precision/recall, and directed F1. Multiple confidence tiers are evaluated (high-only, high+mid, all tiers). Three replicate zero-temperature runs per test case document variability. Citation grounding is assessed by verifying that quoted text matches retrieved abstracts literally.
6. Reproducibility: System is implemented with a browser interface and typed backend pipeline executed both interactively and in headless batch mode. The evaluation uses frozen PubMed corpus snapshots with retrieval cutoff dates to avoid data leakage. Specific Anthropic Claude-sonnet-4-6 model version and configuration details (temperature=0) are fixed. No training code or weights are produced, but the system preserves artifact versions and provenance metadata for audit. Example: For a submitted study with variables representing exposure, outcome, and covariates, DAGForge normalizes each concept, expands query terms via UMLS atoms, retrieves relevant PubMed abstracts using a tiered querying scheme, extracts causal relation snippets from sentences, and prompts the LLM to produce a detailed judgment for each variable pair including an explicit citation and rationale. These judgments are then combined under constraints to generate a DAG with edge directions checked for temporal consistency and acyclicity. The interface presents the graph with clickable evidence cards showing supporting abstracts and exact quoted texts, enabling expert review and iterative refinement.

Technical innovations

• Integration of ontology-aware concept normalization, tiered PubMed literature retrieval, LLM-based pairwise causal reasoning with verbatim evidence grounding, and constraint-checked DAG assembly into a single auditable pipeline.
• Use of a staged, provenance-preserving workflow that freezes a literature snapshot per run, enabling verifiable evidence trails for every causal edge rather than opaque LLM-generated graphs.
• Deterministic enforcement of temporal ordering, exogeneity, and acyclicity constraints during graph assembly combined with LLM reasoning to reconcile local judgments within global graph structure.
• Provision of alternative DAGs via one-edge reversals of ambiguous edges and identification of candidate latent confounders supported by partial evidence for expert inspection.
• Interface design exposing detailed per-edge confidence, provenance, rationale, and exact citation snippets for transparent model-assisted causal graph authoring.

Datasets

• sDAG synthetic cohort — 7 compact biomedical DAGs with known reference graphs — internally constructed synthetic benchmark
• Literature cohort (A, B, C) — 3 biomedical study-derived DAGs from published epidemiological research (Boyle et al. 2015; Reiner et al. 2016; Evandt et al. 2017) — publicly cited, curated DAGs from literature

Baselines vs proposed

• LLM direct baseline (Claude-sonnet-4-6, temp=0): graph directed F1 on literature cohort cold = 0.78 vs DAGForge 0.664 (higher precision, lower recall)
• LLM direct baseline (with study context): graph directed F1 = 0.80 vs DAGForge 0.714
• LLM direct baseline citations exactly matching retrieved abstracts = 0/211 vs DAGForge citations 100% verbatim matched
• Pairwise skeleton F1 on synthetic DAGs: LLM direct baseline unreported vs DAGForge 0.944
• Graph directed F1 on synthetic DAGs: LLM direct baseline unreported vs DAGForge 0.895

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21859.

Fig 1: DAGForge system flow. LLM calls occur inside bounded backend stages; the interface renders persisted

Fig 2: DAGForge interface. The left panel shows the study-submission form with temporal constraints. The right

Limitations

• Literature evidence limited to PubMed abstracts, excluding full-text data and possibly missing relevant causal evidence especially for specialized or newly relevant variables.
• LLM causal judgments depend on model priors and retrieval quality, potentially overstating familiar relations and conflating direct, total, and mediated effects despite provenance.
• The combinatorial explosion of concept pairs entails quadratic cost in retrieval and LLM calls, limiting scalability for large variable sets without future optimizations.
• Evaluation benchmarks reflect reconstruction of curated reference DAGs, which may be parsimonious and incomplete; thus performance metrics do not fully capture causal correctness.
• No controlled user study yet conducted on the interactive interface’s impact on expert workflow and DAG quality improvements.
• Graph assembly relies on heuristic mediation policies and may require more structure-aware LLM passes in future to resolve ambiguous edges.

Open questions / follow-ons

• How to extend retrieval beyond PubMed abstracts to include full-text articles or clinical records for richer evidence coverage?
• Can graph-level context be integrated more tightly into pairwise LLM causal judgment to improve precision without losing recall?
• What are optimal user interface designs and workflows that maximize expert efficiency and correct model biases in interactive DAG curation?
• How to scale DAGForge’s approach to high-dimensional causal graph problems with hundreds or thousands of variables efficiently?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, DAGForge illustrates an advanced example of combining retrieval-grounded large language model reasoning with formal constraint solving and provenance tracking to produce auditable, evidence-linked graph artifacts. Methodologically, it offers a model for how LLM outputs can be bounded and made transparent via staged workflows, citation grounding, and deterministic validations—a principle potentially adaptable to building trustworthy, inspectable AI components in security systems. The focus on combining noisy NLP outputs with domain constraints can inspire bot-detection approaches that fuse probabilistic reasoning and symbolic rules. Furthermore, their detailed provenance and audit mechanisms highlight effective strategies for explainability and validation critical in adversarial settings.

Cite

@article{arxiv2607_21859,
  title={ DAGForge: Auditable Causal DAG Authoring with Biomedical Literature },
  author={ Yi-han Sheu and Michael R. Steigman and Yu Zhou and Bo Wang and Fan-Yu Yen and Jordan W. Smoller },
  journal={arXiv preprint arXiv:2607.21859},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21859}
}

Read the full paper

• arXiv:2607.21859 (abstract)
• arXiv:2607.21859 (PDF)