Less is More: Quality-Aware Training Data Selection for Scientific Summarization

Source: arXiv:2606.24828 · Published 2026-06-23 · By Maria Nefeli Paraskevopoulou, Tatiana Passali, Grigorios Tsoumakas

TL;DR

This work addresses two key challenges in scientific long-document summarization: the limited scale and structure of existing datasets and the variable quality of author-written abstracts used as reference summaries. The authors construct and release PMC-Large, a large biomedical and life science summarization dataset containing 1.88 million PubMed Central articles with Markdown-serialized, section-preserving full text paired with author abstracts. They analyze reference quality on a 10,000-article subset using multiple source-grounded and model-based metrics, revealing substantial variation in alignment and factuality of abstracts relative to the full text. Leveraging these quality signals, they apply training-data selection to fine-tuning of a Qwen2.5-3B model. Models trained on smaller high-quality subsets consistently outperform those trained on larger randomly sampled subsets on factuality-focused metrics. Their results underline that not all abstracts serve as equally reliable supervision, and quality-aware selection can boost training efficiency and summary quality for scientific long-document summarization.

Key findings

• PMC-Large dataset contains 1.88 million biomedical and life science articles from PMC with Markdown-encoded section hierarchy, significantly larger than existing datasets like PubMed (133K articles).
• On a 10K article subset, author-written abstracts show variable reference quality: AlignScore median 0.626 (bottom decile <0.473), G-Eval median 0.900, FineSurE mean 0.850 with bottom decile <0.667.
• Quality metrics show low inter-metric correlation (e.g., SummaC vs AlignScore ρ=0.067), indicating no single metric fully captures abstract quality.
• Fine-tuning Qwen2.5-3B on 1,000 highest-quality examples selected by aggregate of 4 metrics outperforms training on 1,000 random samples (Overall score 0.806 vs 0.361) and matches or exceeds training on full 5,000-sample pool (0.806 vs 0.583).
• Quality-selected subsets scaled to top AlignScore-ranked 1K to 10K from 100K pool outperform random subsets on overall score across factuality metrics, e.g. AlignScore 5K subset achieves 0.679 vs random 5K 0.179.
• Two-stage selection (fast AlignScore pre-filter + multi-metric reranking) further improves performance at scale, with AGGREGATE-ALL 5K model reaching overall score 0.839.
• Zero-shot prompting baselines underperform fine-tuned models on AlignScore, G-Eval, and FineSurE metrics but achieve highest SummaC (0.462), suggesting difference in metric sensitivity to summary style.

Threat model

The adversary is implicit: variable quality of author-written abstracts can act as noisy or misleading supervision and evaluation references, potentially causing summarization models to generate hallucinated or factually incorrect summaries. The adversary can be understood as uncontrolled noise in training data rather than an active attacker. The study assumes no malicious data poisoning or direct adversarial manipulation, focusing on assessing and mitigating inherent quality variance in real-world scientific summarization corpora.

Methodology — deep read

1. Threat Model & Assumptions: The study focuses on improving scientific long-document summarization trained on author abstracts; the adversary model is indirect, concerning how variable quality supervision affects model reliability and factuality. It assumes no direct adversarial attacks but tests robustness against noisy or weak supervision signals.

2. Data: They build PMC-Large from PubMed Central Open Access corpus, consisting of 1.88 million biomedical articles licensed under CC BY and CC0. Each article's body is normalized into Markdown with section hierarchies, filtering out non-narrative sections, figures, tables, and duplications of abstracts in the body. English-only articles with body length (3K-9K tokens) and abstract length (190-440 tokens) between the 10th and 80th percentiles are retained.

3. Architecture/Algorithm: The base model used for supervised fine-tuning is Qwen2.5-3B, a large language model capable of handling long inputs. The fine-tuning prompt inserts the full Markdown-encoded article body into a fixed prompt template requesting a summary. Supervision targets are the corresponding author-written abstracts.

Quality selection uses multiple source-grounded and model-based metrics: AlignScore (alignment via matching), FineSurE (sentence-level factuality errors), SummaC-ZS with PubMedBERT NLI backbone, and G-Eval (LLM-based judgment on coherence, factuality, relevance). Each metric scores article-abstract pairs to estimate reference quality.

4. Training Regime: Fine-tuning involves training the Qwen2.5-3B model for one epoch on various selected subsets of PMC-Large, varying from 1K to 10K examples depending on the experiment scale. Training details including batch size, optimizer, and other hyperparameters are in Appendix B (not fully detailed). The training is performed on disjoint sets distinct from the 10K analysis and 500-example test sets.

5. Evaluation Protocol: Evaluations are performed on a held-out 500-example test set disjoint from training and reference-quality analysis sets. Models are assessed via multiple source-grounded metrics (AlignScore, FineSurE, SummaC, and G-Eval) comparing generated summaries to the article body rather than reference similarity. FineSurE success rate (percentage of successfully parsed outputs) is reported. An overall score aggregates ranks across the four metrics to form a balanced assessment.

Baselines include zero-shot, one-shot, and two-shot prompting and training on random subsets of comparable sizes for fair comparison to quality-selected sets.

6. Reproducibility: PMC-Large is publicly released on Hugging Face. Selection and evaluation metrics are primarily automatic and include public tools. The Qwen2.5-3B model is publicly known. Exact training code, weights, and full hyperparameter details are referenced but not explicitly linked in the paper, suggesting partial but accessible reproducibility. Large-scale experiments on 1.88M articles were limited by compute, with detailed results reported on smaller but representative subsets.

Example Pipeline End-to-End: To train a summarization model on a quality-selected subset, the authors first score 5,000 candidate article-abstract pairs by AlignScore, FineSurE, SummaC, and G-Eval. They aggregate the percentile ranks of these metrics per example and select the top 1,000 highest aggregate scorers. These 1,000 examples have Markdown-formatted articles paired with abstracts as targets. Fine-tuning of Qwen2.5-3B with the fixed prompt is done for one epoch. Evaluation on the 500-example test set shows improved source-grounded factuality metrics compared to models trained on 1,000 randomly sampled examples.

Technical innovations

• Release of PMC-Large, a 1.88M article biomedical long-document summarization dataset with Markdown-preserved, hierarchical section structure optimized for long-context LLMs.
• Comprehensive large-scale analysis of author-written abstract quality in scientific articles using diverse source-grounded and model-based metrics highlighting quality variability.
• Novel training-data selection approach combining multiple factuality and alignment metrics to identify high-quality supervision samples, improving sample efficiency and factuality.
• Demonstration that a two-stage data selection pipeline (fast scalable AlignScore pre-filter followed by re-ranking with multiple metrics) efficiently scales quality-aware subset selection to 100K example pools.

Datasets

• PMC-Large — 1,884,517 article-abstract pairs — PubMed Central Open Access corpus, CC BY/CC0 licensed

Baselines vs proposed

• Zero-shot prompting: Overall score = 0.444 vs aggregate 1K selected subset = 0.806 (higher is better)
• Random 1K sample: Overall = 0.361 vs aggregate 1K selected = 0.806
• All 5K random subset: Overall = 0.583 vs aggregate 1K selected = 0.806
• AlignScore 5K selection (from 100K pool): Overall = 0.679 vs random 5K = 0.179
• All 100K random pool: Overall = 0.429 vs AlignScore 5K selection = 0.679
• Aggregate-ALL 5K selection (two-stage) achieves best Overall = 0.839 across large-scale experiments

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.24828.

Fig 1: and Table 2 summarize the reference-

Fig 2: Token-count distributions for article bodies

Fig 3 (page 11).

Limitations

• Use of automatic metrics for reference quality evaluation, which may not capture all nuanced dimensions of summary quality; no human evaluation is reported.
• Experiments and quality scoring performed on subsets of the full 1.88M dataset due to computational costs; results may not fully extrapolate to entire corpus scale.
• Single base model architecture (Qwen2.5-3B) and one primary fine-tuning setup limits generality of findings across different models or training regimes.
• Dataset limited to English PMC Open Access articles meeting specific license and length criteria, possibly limiting domain or linguistic diversity coverage.
• SummaC metric showed weaker correlation and predictive utility, indicating some selection metrics may be less suited for biomedical long documents.

Open questions / follow-ons

• How do human evaluations of summary quality and faithfulness align with the proposed automatic reference-quality metrics?
• Can quality-aware training-data selection methods generalize to other domains or to different base model architectures and scales?
• What training strategies or loss functions further amplify benefits from high-quality supervision subsets beyond simple example selection?
• How does quality-aware supervision influence model calibration, error modes, or hallucination rates in deployed scientific summarization?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the paper's core insight is that training data quality significantly affects downstream model reliability and factual consistency. In contexts where language models are used to generate or evaluate textual content under adversarial conditions, understanding and selecting high-quality supervision data can reduce the risk of ungrounded or misleading outputs. This complements bot-detection by enhancing the factual grounding of summarization or summarization-like components in content moderation or information verification systems.

Moreover, the multi-metric, two-stage data filtering approach demonstrates a scalable method to improve training efficiency by avoiding large noisy datasets, which aligns with best practices for robust model design in security-sensitive applications. While not directly addressing bot defense, the methodologies and findings offer transferable lessons on leveraging source-grounded metrics and quality-aware training that could enhance model trustworthiness in related NLP tasks within security and fraud detection pipelines.

Cite

@article{arxiv2606_24828,
  title={ Less is More: Quality-Aware Training Data Selection for Scientific Summarization },
  author={ Maria Nefeli Paraskevopoulou and Tatiana Passali and Grigorios Tsoumakas },
  journal={arXiv preprint arXiv:2606.24828},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.24828}
}

Read the full paper

• arXiv:2606.24828 (abstract)
• arXiv:2606.24828 (PDF)