AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

Source: arXiv:2607.11849 · Published 2026-07-13 · By Lingkai Kong, Zijian Wu, Yuzhe Gu, Haiteng Zhao, Wenyong Huang, Shuang Sun et al.

TL;DR

AdvancedMathBench addresses a critical gap in evaluating large language models (LLMs) on advanced mathematical reasoning, particularly natural-language proof generation and verification. Existing benchmarks largely focus on final-answer correctness in high-school or olympiad-level math, lacking rigorous process-level verification and coverage of advanced topics. To overcome this, the authors curate a new benchmark suite with 245 undergraduate and doctoral qualifying-exam level proof problems (ProverBench) and 888 annotated model-generated proof trajectories (VerifierBench). A novel automatic verification pipeline is developed, trained on large-scale expert annotations, to provide both correctness verdicts and granular error analysis of proofs. Benchmarking state-of-the-art proprietary and open-source LLMs reveals substantial room for improvement: the best model achieves only 64.5% and 48.9% correctness on the UG and QE subsets, and verification F1 scores peak at 65.1%, highlighting challenges in advanced proof construction and error detection. The work establishes a rigorous foundation for evaluating advanced mathematical proof capabilities, focusing on the full reasoning process rather than just final answers.

Key findings

• ProverBench includes 245 problems: 200 undergraduate (UG) and 45 doctoral qualifying-exam (QE) level, covering multiple math sub-disciplines.
• GPT-5.5-xhigh achieves a pessimistic verification score of 64.5% on UG and 48.9% on QE splits for proof generation, indicating a significant drop in performance on more advanced problems.
• VerifierBench contains 888 model-generated proofs with expert ground truth labels for process-level verification including error localization and rationale alignment.
• Best proof verification model, DeepSeek-V4-Pro, attains a Meta-Verification Balanced F1 of only 65.1%, revealing difficulty in precise error detection.
• Meta-Verification evaluation shows average true negative rates (TNR) are low (e.g., highest TNR is 55.8%), pointing to a major limitation in rejecting invalid proofs.
• Automatic verification pipeline trained with RL and expert annotations achieves 82.1% Balanced F1 on held-out examples, outperforming LLM-as-judge baselines by over 10 F1 points.
• Ablations show that combining rationale-aware reward, positive-sample augmentation, extra annotation, and pessimistic verification substantially boosts verification robustness and conservativeness.
• Models tend to over-accept plausible but flawed proofs, with high true positive rates (TPR) but low TNR, indicating conservative error detection remains a bottleneck.

Threat model

The threat model assumes the LLM as the entity generating or verifying proofs, tasked with producing rigorously checkable mathematical arguments. The adversary is the model's intrinsic limitations rather than an external attacking adversary. The evaluator assumes no access to formal proof checkers and evaluates natural-language proofs without adversarial tampering. It cannot account for malicious manipulations or proof obfuscations beyond plausible reasoning failures.

Methodology — deep read

The core methodology involves constructing two complementary benchmarks—ProverBench for proof generation and VerifierBench for proof verification—targeting advanced mathematics problems requiring full natural-language proofs.

1. Threat Model & Assumptions: The adversary is primarily the language model being evaluated; it must generate or verify proofs from problem statements without access to formal proof assistants. The focus is on whether models can produce or assess rigorous, checkable proofs, not on adversarial attacks or evasions.

2. Data Curation: Problems are sourced from four main areas—undergraduate exams from top Chinese universities, doctoral qualifying exams from top global schools, official contest problems (S.-T. Yau Competition), and textbook exercises. These are parsed from PDFs into a structured JSON format. Answer-oriented formats like multiple choice or true/false are filtered out. Difficulty pre-screening uses verifier uncertainty (entropy of repeated LLM proof verification results) to remove easy or trivially judged problems and to retain challenging samples. PhD-level mathematical experts perform quality control on problem clarity, correctness, and proof trajectory validity, resulting in 245 problems for ProverBench and 888 proof-problem pairs for VerifierBench.

3. Architecture & Algorithm: A dedicated automatic verification pipeline is trained using ~2,000 proof trajectories with large-scale expert annotations. The verifier outputs correctness verdicts, fine-grained proof quality assessments, error localization, and verification rationales. A meta-verifier model (gpt-oss-120b) is used to evaluate model verification outputs against expert ground truth, scoring validity judgments, error coverage, and rationale quality. Reinforcement learning (GRPO) with meta-verifier reward is applied to train the verifier to produce mathematically grounded analyses rather than binary verdicts alone.

4. Training Regime: Training starts from Intern-S2-Preview-35B checkpoint, using expert-annotated positive and negative samples. Positive-sample augmentation is performed by using corrections to fix erroneous proofs and re-verify them with strong models (DeepSeek-V4-Pro, GPT-5.5), reducing bias towards invalid examples. Pessimistic verification requires unanimity over multiple independent verification passes (8 parallel checks) to accept a proof as valid, increasing conservativeness.

5. Evaluation Protocol: Proof generation is scored pessimistically—the generated proof is accepted only if all independent verification passes confirm validity. Verification is evaluated both roughly on binary validity and more strictly on rationale agreement with meta-verifier scores aligned with expert labels. Metrics include balanced F1, true positive rate, true negative rate, and annotation-level agreement. Models evaluated include proprietary LLMs (GPT-5.5 series, Claude-Opus, Gemini) and open-source models (DeepSeek, Qwen, Kimi, GLM, etc.) with temperature=1.0 and large max token limits.

6. Reproducibility: The paper mentions use of large-scale expert annotation and a multi-stage curation pipeline but does not specify whether code or datasets are publicly released. Model weights for verifiers are not stated as publicly available. The meta-verifier gpt-oss-120b is an LLM used for evaluation but not described as open-sourced.

An example workflow: A problem from the doctoral qualifying exam set is input to a LLM (e.g., GPT-5.5-xhigh) with a prompt to generate a full natural-language proof. The generated proof is fed to the automatic verifier, which runs multiple independent verification passes evaluating each proof step and concluding whether the proof is valid, identifying errors if present. Verification output including rationale and error localization is scored by the meta-verifier against expert annotations, yielding final balanced F1 scores reflecting rigorous process-level evaluation beyond correctness of final answers.

Technical innovations

• A novel, expert-aligned automatic verification pipeline trained on large-scale annotated LLM-generated proof trajectories enabling fine-grained process-level verification beyond binary correctness.
• Use of verifier uncertainty to pre-screen problems and proof trajectories for difficulty, focusing the benchmark on hard and informative samples.
• Meta-verification framework that compares model-generated verification rationales to expert ground truth via a dedicated meta-verifier model for more reliable evaluation of proof verification quality.
• Reinforcement learning with meta-verifier reward to train verifiers that output mathematically grounded and error-sensitive verification analyses instead of only polarity judgments.
• Pessimistic verification protocol requiring unanimous agreement across multiple independent verification attempts to accept proofs, improving conservativeness against subtle proof errors.

Datasets

• ProverBench — 245 problems (200 UG + 45 QE) — sourced from Chinese university undergraduate exams, global doctoral qualifying exams, S.-T. Yau Competition, textbooks
• VerifierBench — 888 problem-proof-ground truth triplets — model-generated proofs with expert annotations

Baselines vs proposed

• GPT-5.5-xhigh: ProverBench UG score = 64.5% vs QE score = 48.9%
• Claude-Opus-4.8: ProverBench UG = 59.0% vs QE = 40.0%
• DeepSeek-V4-Pro: VerifierBench Meta-Verification Balanced F1 = 65.1%
• GPT-5.5-xhigh LLM-as-judge baseline: held-out verification Balanced F1 = 70.6% (rough), 61.6% (meta-verification) vs automatic verifier pipeline 82.1% (rough), 73.9% (meta-verification)
• Positive-sample augmentation increases Meta-Ver TPR from 56.8% to 81.4% and Balanced F1 from 60.5% to 72.2%
• Pessimistic verification raises Meta-Ver TNR from 64.9% to 69.1% and Balanced F1 from 72.2% to 73.9%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11849.

Fig 1: AdvancedMathBench exposes complementary weaknesses in advanced mathematical proof

Fig 3: Subject distributions of ProverBench. The UG split of 200 samples covers core undergraduate

Fig 2: Overview of the AdvancedMathBench methodology. The schema summarizes problem sourcing

Limitations

• Samples are limited to 245 problems for proof generation and 888 proof-verification pairs, which may not cover all domains of advanced mathematics comprehensively.
• Evaluation focuses on English natural-language proofs but does not integrate formal proof assistant languages or formal verification.
• No explicit adversarial robustness testing against worst-case or manipulated proofs is discussed.
• The expert annotation and verification pipeline rely on domain expert availability and quality which may be a bottleneck to scaling.
• Code and datasets availability for reproducibility are not explicitly stated.
• Models evaluated do not include more recent or future LLMs beyond GPT-5.5 or comparable, which may advance capabilities.

Open questions / follow-ons

• How can advanced LLMs better detect subtle logical errors and gaps in complex proof trajectories without formal language constraints?
• What hybrid approaches combining natural-language reasoning with formal proof assistants could improve verification robustness and scalability?
• Can process-level verification techniques be generalized to other domains requiring multi-step complex reasoning beyond mathematics?
• How might continuous or interactive proof generation and verification workflows improve over the static benchmarks provided?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, AdvancedMathBench highlights the current limitations of state-of-the-art LLMs in rigorous advanced reasoning tasks like natural-language mathematical proof generation and verification. While today's LLMs can generate plausible mathematical conclusions, their difficulty in constructing and critically verifying complete proofs reveals a significant weakness exploitable in defense mechanisms. CAPTCHAs or bot defenses requiring multi-step logical reasoning, especially in specialized content domains, could leverage insights here to design challenges that are out of reach for current automated bots based on LLMs. Furthermore, the meta-verification and error-localization techniques pioneered here could inspire more robust automated validation frameworks for complex challenge-response interactions, ensuring not only correct final answers but correct reasoning processes. However, the substantial gap remaining even for top-tier LLMs underscores the necessity for rigorous process evaluation in any advanced challenge system design.

Cite

@article{arxiv2607_11849,
  title={ AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification },
  author={ Lingkai Kong and Zijian Wu and Yuzhe Gu and Haiteng Zhao and Wenyong Huang and Shuang Sun and Zhicheng Xiong and Xiaotian Zhang and Shuya Zhao and Yan Wang and Disheng Xu and Wenwei Zhang and Kai Chen },
  journal={arXiv preprint arXiv:2607.11849},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11849}
}

Read the full paper

• arXiv:2607.11849 (abstract)
• arXiv:2607.11849 (PDF)