AMTFV: Agentic Mathematical Tool-Flow Verification for LLM Self-Correction

Source: arXiv:2607.29549 · Published 2026-07-31 · By Rui Zou, Yutao Zhu, Mengqi Wei, Ji-Rong Wen

TL;DR

This paper addresses the challenge of reliably verifying and self-correcting mathematical problem solutions generated by large language models (LLMs). Traditional approaches either employ natural-language reflection for self-correction, which lacks exact computational rigor, or generate executable verification code prematurely tightly coupling modeling and implementation, making verification fragile and error-prone. AMTFV (Agentic Mathematical Tool-Flow Verification) introduces Mathematical Tool Flow (MTF), an interrupt-execute-resume interface that cleanly separates high-level verification modeling by an LLM agent from concrete exact computation handled by a specialized mathematical toolbox. This decoupling allows the LLM to specify "what" to verify in explicit mathematical terms without managing low-level execution details. The toolbox agent translates MTF requests into calls on symbolic and exact computation packages (e.g., SymPy), executes them, and returns results that guide answer adjudication, iterative self-correction, and verification workflow revision.

AMTFV is evaluated on five challenging mathematical reasoning datasets (AIME 2024/2025, BRUMO 2025, HMMT February 2025, AMO Bench) across seven base-model configurations from DeepSeek, GPT, and Gemini families. It significantly outperforms natural-language reflection, state-of-the-art self-correction, and program-driven verification baselines such as ProgCo, with accuracy improvements up to 8.3 percentage points on average. These gains are especially pronounced on examples requiring medium to high verification complexity involving multiple verification interactions. Detailed analysis shows that AMTFV improves error correction (initially wrong to correct) rates substantially while reducing regression on initially correct answers. The work highlights the benefits of explicit verification-target modeling and reliable exact computation delegation to specialized tools for robust LLM self-correction in mathematical problem solving.

Key findings

• AMTFV improves average final-answer accuracy by up to 8.3 percentage points over strongest baseline (ProgCo-family) on DeepSeek-Flash model configuration.
• AMTFV raises DeepSeek-Flash final accuracy from 44.1% to 61.2%, DeepSeek-Pro from 38.8% to 57.1%, and DeepSeek-Flash-Think from 88.8% to 89.4%.
• AMTFV achieves higher average accuracy than ProgCo and ProgCo-Py across all seven tested base-models, with gains ranging +0.6 to +8.3 percentage points.
• AMTFV attains 18.4% error-correction rate (incorrect to correct) on 1190 model–problem pairs vs 9.4% for ProgCo and 12.2% for ProgCo-Py.
• AMTFV reduces error-introducing regression (correct to wrong) to 2.8%, lower than 3.7% for ProgCo and 4.2% for ProgCo-Py.
• Accuracy gains concentrate strongly on samples with medium and high verification complexity (multiple MTF calls), with normalized gains of 39.2% and 84.6% respectively, exceeding ProgCo-Py by 20.0 and 29.2 percentage points.
• MTF decouples verification-target specification from execution, enabling explicit, inspectable, and reusable mathematical verification workflows.
• AMTFV employs a loop of verification agent, answer-revision agent, and verification-workflow revision agent, orchestrated through the MTF interface.

Threat model

The adversary is the inherent imperfection of LLM-generated mathematical answers which may contain computational errors, flawed reasoning steps, or incomplete verification. The threat arises from models' inability to self-verify reliably without external tool feedback. The system assumes a benign environment where the toolbox backend and agents operate correctly; no adversarial attacks or malicious interference are considered.

Methodology — deep read

The paper presents AMTFV, an autonomous framework to verify and self-correct mathematical answers from LLMs by decoupling verification modeling from execution via an intermediate Mathematical Tool Flow (MTF) interface.

1. Threat Model & Assumptions: The adversary is implicit — typical LLMs prone to computational errors, reasoning flaws, or incomplete verification. The system assumes no external verification feedback but relies on exact computation tools for trustworthy verification. It does not consider adversarial manipulation or malicious actors explicitly.

2. Data: The evaluation uses five challenging mathematical problem datasets totaling 170 problems: A24 (AIME 2024, 30 problems), A25 (AIME 2025, 30 problems), B25 (BRUMO 2025, 30 problems), HMM (HMMT February 2025, 30 problems), and AMO Bench (50 Olympiad problems with diverse formats). These combine multistep reasoning, exact computation, and complex constraints.

3. Architecture / Algorithm: AMTFV decomposes verification into three cooperating agent components: a verification agent (Aver), an answer-revision agent (Aans), and a verification-workflow revision agent (Aflow). These agents iteratively construct and revise explicit verification workflows captured in MTF requests — tuples ⟨context Γ, object M, return operation ρ⟩ representing mathematical objects and computational intents. MTF requests are interrupted from model generation, parsed by a mathematical toolbox agent (Atool) that selects appropriate backend tools (e.g., SymPy for symbolic math, itertools for enumeration, Fraction for exact rational arithmetic), generates executable calls, and executes them in a Python environment. The execution results are fed back to guide adjudication, answer revision, or modification of verification workflows. This interrupt–execute–resume pattern allows the LLM to focus on "what to compute" at a high mathematical level, avoiding premature code generation.

4. Training Regime: The paper focuses on inference-time agent orchestration of existing large language models (DeepSeek variants, GPT-5.4-Mini, Gemini-2.5-Flash-Lite) rather than training new models. Hyperparameters such as max iteration turns (max_turn=3) limit correction rounds. The toolbox backend uses deterministic symbolic/exact computation packages without learning. Specific seeds or hardware details for inference are not explicitly detailed.

5. Evaluation Protocol: Initial candidate answers are extracted from base-model outputs. All methods share identical initial answers for fair comparison. Primary metric is final-answer accuracy aggregated weighted by dataset size. Baselines include natural-language correction methods (reflex, Self-Refine, Self-Reflection, CheckList), reasoning-enhanced CoT-Tool, and verification-enhanced ProgCo and ProgCo-Py (Python-enabled). Iterations are limited to 3 correction rounds. Experiments cover multiple base-model architectures and cross-dataset tests. Further analyses include error correction/preservation transition rates and accuracy stratified by verification complexity (measured by number of MTF calls). Statistical significance or tests are not explicitly mentioned. Reproducibility is supported via public code repository (link provided).

6. Reproducibility: The authors note forthcoming code release under https://github.com/TicusFFF/mathematical-self-correction/tree/main/S2-1_AMTFV. Datasets are public or standard math contest sets. Backends use open-source symbolic math tooling (SymPy, itertools, Fraction). Model checkpoints and prompts are partially referenced from DeepSeek, OpenAI GPT, and Google Gemini APIs but full prompt details are not described.

Concrete Example: For verifying a symbolic expression candidate for a chocolate bar price problem, the LLM constructs an MTF request specifying the symbolic context, the difference expression to check, and the desired operation (simplify). Upon MTF request emission, generation interrupts, the toolbox agent generates a SymPy call to simplify the expression, runs exact symbolic simplification, and returns the result. Seeing a nonzero simplified difference, the verification agent adjudicates failure, triggering answer revision and workflow updates in the next iteration. This loop continues until verification passes or iteration budget is exhausted.

Technical innovations

• Introduction of Mathematical Tool Flow (MTF), an interrupt–execute–resume interface that cleanly decouples high-level verification modeling from exact tool execution.
• A mathematical toolbox agent (Atool) that parses MTF requests, selects from specialized tools (symbolic, enumeration, exact arithmetic), and generates executable calls for backend computation.
• A closed-loop multistage agent framework combining verification, answer revision, and verification-workflow revision agents leveraging MTF results to iteratively improve mathematical answer correctness.
• Operationalization of verification complexity through MTF call counts to analyze method performance relative to verification difficulty.

Datasets

• A24 (AIME 2024) — 30 problems — public math contest dataset
• A25 (AIME 2025) — 30 problems — public math contest dataset
• B25 (BRUMO 2025) — 30 problems — challenging condition and multistep reasoning problems
• HMM (HMMT February 2025) — 30 problems — competition problems across algebra, geometry, combinatorics, number theory
• AMO Bench — 50 problems — Olympiad-level problems with diverse problem and answer formats

Baselines vs proposed

• Initial Score on DeepSeek-Flash: 44.1% vs AMTFV: 61.2%
• Initial Score on DeepSeek-Pro: 38.8% vs AMTFV: 57.1%
• Initial Score on DeepSeek-Flash-Think: 88.8% vs AMTFV: 89.4%
• ProgCo on DeepSeek-Flash: 51.2% vs AMTFV: 61.2%
• ProgCo-Py on DeepSeek-Flash: 52.9% vs AMTFV: 61.2%
• AMTFV improves over ProgCo family by +8.3, +1.2, and +7.7 percentage points respectively on DeepSeek-Flash, DeepSeek-Flash-Think, and DeepSeek-Pro.
• AMTFV outperforms ProgCo and ProgCo-Py in average accuracy across all seven base-model configurations with gains up to +8.3 percentage points.

Limitations

• Evaluation focuses on mathematical contest datasets with relatively small sample size (170 problems across five sets).
• The approach does not evaluate adversarially crafted problems or explicitly consider malicious model or tool behavior.
• No detailed ablation studies quantifying the exact contribution of each of the three agents (verification, answer revision, workflow revision) independently.
• Unclear how sensitive performance is to iteration budget, model prompt design, or toolkit quality.
• The framework depends on access to reliable mathematical tools (e.g., SymPy) and assumes their correctness and availability during inference.
• Reproducibility may be challenged by partial reliance on proprietary base models (OpenAI GPT, Google Gemini) with undisclosed weights or training data.

Open questions / follow-ons

• How well does AMTFV generalize to larger-scale mathematical problems or domains beyond contest-style math (e.g., formal theorem proving, scientific computations)?
• Can the framework incorporate learning-based tool selection or adaptivity to optimize verification workflows dynamically?
• What is the impact of different base LLM capabilities or prompt engineering strategies on AMTFV effectiveness?
• How can AMTFV be extended to multi-agent collaborative verification or applied to non-mathematical domains requiring exact computation and verification?

Why it matters for bot defense

For bot-defense and CAPTCHA design, AMTFV demonstrates a principled approach to verifying complex multi-step reasoning outputs from LLMs through decoupled, tool-augmented exact computation. This separation between verification modeling and reliable backend execution reduces brittle code-generation errors common in program-driven verification approaches, improving overall verification robustness. Practitioners designing CAPTCHAs or bot-detection challenges reliant on LLM mathematical reasoning can apply similar architectural principles to better adjudicate solution correctness, enable iterative answer refinement, and handle complex verification objectives.

In adversarial bot-defense contexts, decoupling verification intent specification from execution tooling allows robust validation without exposing low-level implementation details that could be reverse-engineered by bots. The MTF paradigm may inspire layered verification interfaces where the challenge verifier fully controls exact computations while the reasoning agent focuses on high-level math logic, helping to detect and prevent automated solver attacks on math-based CAPTCHAs or logic puzzles. The closed-loop refinement agents and workflow revision strategies also suggest frameworks for continuous bot-challenge improvement based on detected verification weaknesses.

Cite

@article{arxiv2607_29549,
  title={ AMTFV: Agentic Mathematical Tool-Flow Verification for LLM Self-Correction },
  author={ Rui Zou and Yutao Zhu and Mengqi Wei and Ji-Rong Wen },
  journal={arXiv preprint arXiv:2607.29549},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.29549}
}

Read the full paper

• arXiv:2607.29549 (abstract)
• arXiv:2607.29549 (PDF)