Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation
Source: arXiv:2607.29250 · Published 2026-07-31 · By Goutham Ramakrishnan, Megha Sharma
TL;DR
This paper addresses the critical challenge of training small language models (SLMs) for function-calling tasks, where training data scarcity and quality limit model performance. Unlike large language models, SLMs cannot compensate for noisy or insufficient supervision via scale, making data generation methods essential. To address this, the authors introduce Data Turnstile, an open-source, scalable framework for generating high-quality synthetic function-calling data by decomposing multi-turn interactions into modular, role-based generation steps with validation and error-feedback loops. This approach improves data quality, structural correctness, and diversity, enabling efficient fine-tuning of SLMs.
Empirical evaluation on both single-turn (BFCL) and multi-turn (τ2-bench) benchmarks demonstrates that SLMs fine-tuned on Turnstile data substantially outperform baselines, including larger models without specialized fine-tuning. For example, a Qwen3-0.6B model fine-tuned on Turnstile data achieves 75.9% accuracy on BFCL single-turn (vs. 67.4% base), nearly matching Qwen3-4B (7× larger). On the multi-turn Telecom domain, Turnstile-trained Qwen3-1.7B achieved 31.1% pass rate, surpassing zero-shot Qwen2.5-32B (19× larger). The authors also release a dataset of 100K+ multi-turn interactions spanning 1,000+ APIs.
Key findings
- Qwen3-0.6B fine-tuned on Turnstile data achieves 75.9% overall accuracy on BFCL single-turn function calling, exceeding its 67.4% base model score and approaching Qwen3-4B's 79.9%.
- Turnstile-generated datasets show +15.3 percentage point accuracy improvements over baseline open-source datasets (Raw-OS: 55.1% vs Turnstile-OS: 70.4%) on BFCL single-turn.
- Turnstile out-of-distribution data (from synthetic domains) yields +2.5pp gain over Turnstile-OS on BFCL, demonstrating improved generalization to unseen APIs.
- Adding in-distribution BFCL-specific API data provides an additional +3.0pp gain (72.9% -> 75.9%) on BFCL single-turn.
- Qwen3-1.7B fine-tuned on Turnstile Telecom multi-turn data achieves 31.1% passˆ1, a 4.7× improvement over its 6.6% baseline and surpassing Qwen2.5-32B-Instruct (27.4%).
- Qwen3-0.6B achieves 24.6% passˆ1 on Telecom multi-turn, a 7× increase over 3.5% baseline, despite being 53× smaller than 32B models.
- Tool-call weighted supervised fine-tuning improves multi-turn Telecom passˆ1 by +1.6 to +3.9pp over unweighted loss.
- Multi-turn tasks require chain-of-thought (CoT) reasoning for strong performance, with no-think variants dropping 11-22pp; in contrast, CoT is less helpful or detrimental in single-turn.
Methodology — deep read
The authors propose Data Turnstile, a framework designed to generate high-quality synthetic training data for function-calling tasks by decomposing multi-turn conversations into atomic roles connected in a DAG template.
Threat model & assumptions: The adversary is not explicitly defined as this is a data generation and model training method. The framework assumes access to API specifications and no prior high-quality dialogue data. It seeks to provide diverse, correct training interactions without hallucinations or structural errors.
Data: Generation is driven by user-defined APIs and domain policies. They construct datasets from multiple API sets including open-source (xLAM, Glaive), synthetically generated domains, BFCL single-turn APIs, and Telecom domain-specific APIs from τ2-bench. Dataset sizes vary from ∼36K to >200K interactions. Generated data contains multi-turn conversations with five role types: USER, API CALL, API OBS (tool output), ASSISTANT, and THINKING (chain-of-thought reasoning traces).
Architecture/Algorithm: Turnstile uses template-based generation where an interaction is a DAG of roles. Each role is generated independently conditioned on template context and previous role outputs, enabling role-wise validation (schema compliance, parameter grounding, hallucination checks). Generation uses a supervised large model (Qwen2.5-32B-Instruct) as teacher LLM, served efficiently via vLLM. If any role fails validation, it is retried with error feedback or aborted early to avoid cascading errors. Templates encode structural ordering constraints and API-specific rules. User personas and dynamic perturbations (e.g., API failures, information requests) increase diversity and robustness.
Training regime: They fine-tune Qwen3 models (0.6B, 1.7B, 4B) on the synthetic data using AdamW optimizer, learning rate 5e-5, linear warmup for 50 steps, batch size 64, and max sequence length 4096 (single-turn) or 8192 (multi-turn). Models are trained in 'think' mode with CoT enabled and 'no-think' direct output modes for ablation.
Evaluation: Two benchmarks used - BFCL for single-turn function calling (evaluating AST parses of generated calls) and τ2-bench Telecom domain for multi-turn agentic workflows (evaluating passˆ1 success rate on 114 diagnostic tasks). They compare base models with no SFT, open-source SFT, and Turnstile SFT, including ablations for CoT usage and weighted loss on API call tokens. Results include per-category accuracy and multi-turn performance with statistical confidence intervals.
Reproducibility: The Data Turnstile framework and Synthetic Domains dataset (~100K interactions spanning 1,000+ APIs) are publicly released. Model weights and code beyond generation framework are not mentioned as released. The authors provide detailed dataset statistics, training hyperparameters, and a modular, open-sourced Python package to reproduce data generation.
Example end-to-end: For single-turn data, a template defines a DAG with USER role followed by THINKING (optional), API CALL, API OBS, and ASSISTANT. The teacher LLM generates a realistic user query initiating a function call, next step generates the corresponding API call conforming to schema, followed by tool output generation validated for JSON correctness, and ending with an assistant response. Each step passes validity checks or retries with error feedback. The final dialog is a high-quality training instance for fine-tuning SLMs.
Technical innovations
- Decomposing multi-turn function-calling interactions into a DAG of atomic roles with independent, constrained generation and validation steps to catch errors early and improve data quality.
- Incorporation of role-specific validation and an error-feedback retry mechanism that incrementally improves generation success with smaller, open-weight LLMs instead of relying on proprietary large models.
- Template-based structured diversity control over interaction complexity, API coverage, and edge cases (e.g., tool execution failures, information requests) via dynamic perturbations.
- Weighted supervised fine-tuning loss applied selectively on API call tokens to prioritize critical tool invocation correctness in multi-turn agentic tasks.
Datasets
- Synthetic Domains — ~100K interactions — public release via HuggingFace, 1,025 APIs across 50+ domains
- BFCL single-turn — ~159K interactions — derived from BFCL test suite APIs
- τ2-bench Telecom multi-turn — ~36K interactions — policy-driven Telecom domain data
- xLAM — ~230K interactions — open-source API definitions
- Glaive — ~199K interactions — open-source API definitions
Baselines vs proposed
- Qwen3-0.6B base (think eval): 67.4% overall accuracy on BFCL single-turn vs Turnstile-ODD+ID fine-tuned: 75.9%
- Raw open-source data SFT (Raw-OS): 55.1% accuracy vs Turnstile-OS (same APIs, controlled generation): 70.4% accuracy on BFCL single-turn
- Qwen3-1.7B base (think eval) 78.4% vs Qwen3-0.6B Turnstile-fine-tuned (no-think): 75.9% on BFCL single-turn
- Qwen3-1.7B base on τ2-bench Telecom: 6.6% passˆ1 vs Turnstile SFT: 27.2%, vs Qwen2.5-32B-Instruct zero-shot: 27.4%
- Qwen3-0.6B base on τ2-bench Telecom: 3.5% passˆ1 vs Turnstile SFT: 23.0%
- Qwen3-4B Turnstile SFT with tool-call weighted loss: 38.8% passˆ1 vs unweighted SFT: 36.6% on τ2-bench Telecom multi-turn
- Turnstile SFT no-think (single-turn) outperforms think mode by 3.1pp (75.9% vs 72.8%) on BFCL, reversing trend seen in base models
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.29250.
Fig 1: τ 2-bench multi-turn results: Domain adaptation with Turnstile data helps SLMs beat zero-shot 32B models.
Fig 2: Data Turnstile Overview: role-wise generation with per-step validation and error-feedback based retry.
Fig 3: Multi-turn data generation for τ 2-bench Telecom: reusable Issues compose into Scenarios with dynamic
Limitations
- Though error feedback improves generation success, failure rates increase significantly with interaction complexity (e.g., up to 22% failure with 4 parallel calls).
- No adversarial robustness or attack resistance evaluation included, so the security implications of synthetic training data are unexplored.
- Evaluation is limited to BFCL and τ2-bench Telecom domains; broader generalization to other APIs or real user data not assessed.
- Model and dataset release is partial; the community may face challenges reproducing exact fine-tuning results without access to same teacher LLM or infrastructure.
- The approach assumes access to fully specified and formalized API schemas, which may be unavailable or out-of-date in some practical deployments.
- Chain-of-thought benefits differ substantially between single-turn and multi-turn tasks; the framework does not yet automate optimum CoT usage selection.
Open questions / follow-ons
- How does Turnstile-generated data perform under adversarial or out-of-distribution user queries that attempt to confuse API calls or induce hallucinations?
- Can the error-feedback role-wise generation approach scale efficiently to dialogues with dozens of turns and complex branched API workflows without prohibitively high failure rates or costs?
- What are the trade-offs between using chain-of-thought reasoning during training and inference for small models across diverse function-calling domains?
- How effectively can Turnstile templates and perturbations be automatically derived or learned from real user logs to further improve data realism and coverage?
Why it matters for bot defense
Data Turnstile presents a systematic and modular methodology for producing high-quality, structurally validated synthetic function-calling data that enables small language models to close performance gaps with much larger models. From a bot-defense or CAPTCHA perspective, this approach illustrates how careful data generation can significantly improve the robustness of agentic functionalities built on limited compute budgets. Practitioners building CAPTCHA systems relying on function-calling or interactive challenges could leverage Turnstile-like generation methods to simulate diverse attacker or user interaction patterns, thereby training models that better discern legitimate from malicious scripted API usage.
Moreover, explicit role-wise validation and error-feedback to mitigate hallucinations or malformed API calls align with best practices for minimizing exploitable errors in agentic workflows. The approach also demonstrates the importance of fine-grained control over data diversity, correctness, and interaction complexity — aspects directly relevant when designing human-verification tasks resistant to automation. Finally, the open-source availability of Turnstile and synthetic datasets provides a valuable resource for bot-defense researchers experimenting with function-calling detection or mitigation strategies in constrained model deployments.
Cite
@article{arxiv2607_29250,
title={ Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation },
author={ Goutham Ramakrishnan and Megha Sharma },
journal={arXiv preprint arXiv:2607.29250},
year={ 2026 },
url={https://arxiv.org/abs/2607.29250}
}