Are we facing a reproducibility crises in materials synthesis? A systematic review of Turkevich AuNP synthesis and CVD MoS2 growth

Source: arXiv:2607.14849 · Published 2026-07-16 · By Julia S. Correa, Leandro V. Silva, Nichollas G. G. Silva, Cesar Raitz, Alex S. Lima, Daniel Grasseschi

TL;DR

This paper addresses a critical and underexplored issue in materials science: the reproducibility crisis in synthesis methods, focusing on two widely used and foundational synthesis techniques— the Turkevich method for gold nanoparticle (AuNP) synthesis and chemical vapor deposition (CVD) growth of MoS2. Although these methods are considered standard and reliable within the community, the authors rigorously assess reproducibility through a comprehensive systematic review and meta-analysis of over 1,300 published studies each for AuNP and MoS2. They apply an adapted PRISMA and modified SPIDER framework combined with a hybrid human-AI abstract screening protocol and detailed methodological checklists to evaluate transparency, parameter reporting, and experimental consistency across the literature.

The key finding is that despite extensive literature and a widespread perception of reproducibility, only a very small fraction (around 2-4%) of studies met rigorous standards for methodological transparency needed to support reproducibility claims. Crucial experimental parameters such as precursor concentrations, pH, stirring rate, and temperature measurement were inconsistently or poorly reported. Statistical analyses like reporting replicates and variance were rare, limiting cross-lab comparability. The meta-analysis further highlighted variability and inconsistency in synthesis outcomes related to incomplete reporting. This work reveals that the apparent reproducibility crisis largely stems from insufficient methodological reporting rather than inherent irreproducibility of the syntheses. The paper demonstrates the value of systematic review approaches in materials synthesis to uncover reproducibility gaps and guide future protocol standardization and more transparent reporting practices.

Key findings

• From an initial corpus of ~1,300 articles per synthesis type, only 466 (36%) passed abstract screening for relevance and methodological clarity (Figure 3B).
• Of 457 full-text accessible AuNP synthesis articles, only 19 (4%) met the ≥75% methodological quality threshold required for reproducibility meta-analysis (Figure 3E).
• Critical parameters such as solution pH were reported in only 7% of AuNP studies, despite known major influence on particle size/morphology.
• Stirring rate and reporting of replication experiments were reported in only 7% and 2% of AuNP studies respectively, limiting assessment of batch-to-batch variability.
• Only 34% of AuNP studies correctly reported synthesis temperature by direct solution measurement; many incorrectly cited hot plate set points or boiling point assumptions.
• Statistical analyses such as standard deviations, size distribution histograms, or replicates were explicitly mentioned in only 21% of AuNP abstracts, indicating poor rigor in result reporting.
• For MoS2 CVD synthesis, 38% of studies mentioning reproducibility focused on one-factor optimization rather than full factorial or statistically designed reproducibility studies; only one study reported a comprehensive reproducibility assessment.
• No study in the AuNP dataset achieved perfect methodological reporting score; widespread incomplete parameter disclosure was evident.

Threat model

N/A — this is not a security-focused paper. The implicit 'adversary' is nature and human procedural variability introducing uncontrolled factors, combined with incomplete or inadequate reporting that impedes reproducibility by legitimate scientific researchers attempting independent replication.

Methodology — deep read

The authors conducted a systematic review and meta-analysis with carefully adapted biomedical protocols (PRISMA and SPIDER) tailored to chemical synthesis in materials science. The key steps were:

1. Threat Model/Assumptions: Although not a security paper, the implicit threat model is that extraneous variables and insufficient methodological reporting impede reproducibility, not adversarial interference. The goal was to evaluate publicly reported literature for transparency and consistency.

2. Data Collection: Literature was searched comprehensively in Scopus and Web of Science databases using search queries developed and validated against sentinel landmark papers for Turkevich AuNP synthesis (1999–2024) and MoS2 CVD growth (2010 onward). Initial retrieval included approximately 1,300+ articles per method. Duplicates, non-English, low-quality/predatory journals, and non-primary research articles were excluded systematically.

3. Screening Pipeline: Abstract screening combined a human evaluated STROBE-inspired checklist assessing methodological relevance, clarity, and transparency with a Python-based keyword classification algorithm. The algorithm was iteratively trained and validated against human reviewers until Cohen’s kappa > 0.7 was achieved. Articles scoring ≥75% and passing key exclusion criteria advanced.

4. Methodological Evaluation: Full texts passing abstract screening were assessed by humans with a rigorously defined checklist tailored to each synthesis method assessing reporting completeness of critical parameters—e.g., precursor concentrations, temperature measurement techniques, reagent addition order, pH, stirring rate for AuNPs; precursor masses, reactor geometry, growth time for MoS2 CVD. Binary yes/no items weighted by importance, with ≥75% score threshold for inclusion to meta-analysis. Key exclusion criteria removed modified protocols (e.g., flow reactors), use of commercial nanoparticles, or omission of precursor concentrations.

5. Data Extraction & Meta-Analysis: Quantitative synthesis parameters and reported material outcomes (particle sizes, size distribution for AuNPs; thickness, grain size, layer number for MoS2) were extracted from qualifying studies. Using R and METAFOR package, meta-analyses tested consistency of reported outcomes across studies with similar parameter settings, identifying variability and reproducibility gaps.

6. Reproducibility: The workflow and data/code supporting the screening algorithm are openly available on the Open Science Framework, enhancing transparency. However, no frozen pretrained models or datasets were referenced for replication of the meta-analysis, which relies on published information extraction.

Example end-to-end: For AuNPs, ~1,299 articles were initially collected, 466 passed abstract screening. Among these, 457 accessible full texts were methodologically assessed with 19 articles reaching the ≥75% quality threshold. Parameters such as pH, stirring rate, and temperature measurement were evaluated for completeness to understand reproducibility bottlenecks. Statistical reporting of particle size distributions was also analyzed, revealing substantial underreporting and inconsistent practices impeding reproducibility across labs.

Technical innovations

• Adaptation of biomedical systematic review frameworks (PRISMA and SPIDER) for evaluating reproducibility in materials synthesis, tailored to chemical experimental workflows.
• Development of a hybrid human-and-AI screening pipeline where a Python keyword classification algorithm was iteratively refined and validated vs human evaluators achieving >70% agreement, to efficiently scale screening of thousands of articles.
• Use of a structured STROBE-inspired checklist specifically tailored to quantitatively assess methodological completeness and transparency of materials synthesis parameters affecting reproducibility.
• Large-scale meta-analytic synthesis using the METAFOR R package on literature-extracted quantitative synthesis parameters to statistically evaluate inter-laboratory consistency in nanoparticle size and MoS2 film growth characteristics.

Datasets

• AuNP Turkevich synthesis literature - ~1,300 articles (Scopus and Web of Science) from 1999 to 2024
• MoS2 CVD growth literature - ~1,300 articles (Scopus and Web of Science) from 2010 onward

Baselines vs proposed

• Initial database: ~1,300 articles per method vs final ≥75% methodological quality inclusion: 19 (4%) for AuNPs and only one comprehensive reproducibility study for MoS2 CVD.
• Abstract screening approval rate: 36% for AuNP studies after human + AI hybrid screening.
• Parameter reporting rates: pH reported in 7% of AuNP studies; stirring rate in 7%; replication experiments in 2%; temperature correctly reported in 34%.
• Statistical analysis presence: 21% of AuNP abstracts mention size variance or replicate experiments.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14849.

Fig 1 (page 1).

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Limitations

• The analysis relies on reported literature only; unpublished negative results or grey data are not captured, potentially biasing conclusions.
• The meta-analysis is limited by incomplete and inconsistent parameter reporting in the primary studies, restricting statistical power and interpretability.
• Statistical testing for reproducibility focuses mainly on reported particle size/distribution metrics; characterization variability or measurement errors are not independently verified.
• MoS2 CVD reproducibility assessment is limited due to scarcity of systematic studies and reliance on single comprehensive reports.
• No direct experimental replication was performed to empirically confirm inferred reproducibility conclusions from meta-analysis.
• Use of automated text classification, while validated, may misclassify some abstracts, especially those with ambiguous terminology or incomplete abstracts.

Open questions / follow-ons

• How can the community establish standardized minimum parameter reporting guidelines specifically tailored for nanomaterials synthesis protocols like Turkevich AuNP and MoS2 CVD?
• What is the impact of environmental variables (lab humidity, batch reagent quality) on synthesis reproducibility that go beyond standard documented parameters?
• Can automated natural language processing tools be further enhanced to extract more granular synthesis conditions and experimental outcomes from broader literature corpora?
• How does reproducibility correlate with measurement and characterization standards for materials produced under nominally identical synthesis conditions?

Why it matters for bot defense

For bot-defense engineers developing CAPTCHAs or synthetic interaction detection systems, this paper illustrates the challenges of reproducing complex experimental protocols under variable conditions and incomplete parameter disclosure. Analogously, in bot-detection and CAPTCHA research, transparency in reporting model architectures, training conditions, and evaluation metrics is critical to reproducibility and comparability across defenses. The hybrid human-machine screening methodology may inspire automated literature review or vulnerability assessment pipelines in security domains. More broadly, this work highlights the necessity of standardized, detailed protocol reporting to enable robust replication — a principle equally relevant when designing secure, interpretable CAPTCHA generation or bot-detection systems that must be reproducible and verifiable by the wider security community.

Cite

@article{arxiv2607_14849,
  title={ Are we facing a reproducibility crises in materials synthesis? A systematic review of Turkevich AuNP synthesis and CVD MoS2 growth },
  author={ Julia S. Correa and Leandro V. Silva and Nichollas G. G. Silva and Cesar Raitz and Alex S. Lima and Daniel Grasseschi },
  journal={arXiv preprint arXiv:2607.14849},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14849}
}

Read the full paper

• arXiv:2607.14849 (abstract)
• arXiv:2607.14849 (PDF)