The Significance of Style Diversity in Annotation-Free Synthetic Data Generation

Source: arXiv:2606.20400 · Published 2026-06-18 · By Zahra Abbasiantaeb, Zeno Belligoli, Omar Essam, Mohammad Aliannejadi

TL;DR

This paper addresses the challenge of generating high-utility synthetic data for intent classification in task-oriented dialogue systems entirely without human-annotated seed data. The authors propose a novel annotation-free synthetic dialogue generation framework that relies only on intent definitions. The key innovation is differentiating and incorporating two types of attributes—topic and style—where style refers to linguistic user behavior in dialogues. They introduce two post-hoc style transfer models, Univ and Exam, to diversify LLM-generated utterances toward more human-like styles. An LLM-based filtering step is used to remove low-quality samples. Experimentally, they demonstrate on a large public (SGD) and an industrial dataset (EIC) that their synthetic data achieves up to 93.3% of the accuracy of human-annotated data. A crucial finding is that style diversity plays a far greater role than topic diversity in synthetic data utility by mitigating spurious correlations between style and intent. Incorporating style attributes during generation outperforms post-hoc stylization, indicating that explicit style control in generation is more effective. The stylization models also improve utility by up to 4.7%. These results highlight the importance of modeling linguistic style diversity for producing practical, annotation-free synthetic dialogue datasets in real-world, dynamic industrial settings.

Key findings

• Synthetic data achieves up to 93.3% of human-annotated training data accuracy on the enterprise EIC dataset and 90.7% on the public Schema-Guided Dialogue (SGD) dataset using Llama for intent classification (Table 2).
• Style attributes contribute more significantly than topic attributes to the utility of synthetic data, with the highest G-Vendi utility score of 9.54 compared to 9.21 (no-attribute) and 9.36 (topic-only) on the EIC dataset.
• Filtering generated data using an LLM-based judge improves downstream intent classification performance (Table 4), e.g. boosting F1 from 0.706 to 0.725 on EIC using Llama model.
• Removing style attributes causes a larger utility drop on the more linguistically diverse EIC industrial dataset compared to the academic SGD dataset, highlighting the importance of style diversity in real-world data.
• Stylization models Univ and Exam improve intent classification over no-attribute baselines by up to 4.7% F1 on synthetic data (Table 6), with Exam generally outperforming Univ.
• Incorporating style attributes during generation yields better downstream performance than post-hoc stylization, except in one Llama block SGD case, confirming style control is best done during generation.
• BLEU similarity between synthetic utterances and human data improves by up to 16.67 points after stylization for EIC, showing that stylistic adaptation moves synthetic text closer to realistic domain style.
• Models trained purely on synthetic data exhibit a significant class-wise F1 drop (from 0.84 to 0.38) when mixed with real human data, suggesting models exploit consistent stylistic artifacts to overfit classes.

Threat model

The paper implicitly considers an adversarial setting where no prior human-annotated data for training dialogue intent classifiers is available, akin to bootstrapping a new domain or locale. The approach assumes an adversary with no capability to inject or manipulate annotations but assumes the use of powerful LLMs for data generation and quality control. The threat is unintentional quality degradation or bias in synthetic data that misguides trained models. Adversaries cannot influence the LLMs internally but data must be robust to out-of-domain or stylistic spurious correlations.

Methodology — deep read

The paper tackles annotation-free synthetic dialogue generation for intent classification by using only intent definitions without human-annotated seed utterances. The threat model assumes no prior annotated data, only intent names and definitions known. The adversary is not clearly defined but implicit; the system must produce data useful for training intent classifiers under real-world conditions.

Data provenance includes two main datasets: the public Schema-Guided Dialogue (SGD) dataset (large scale, 20+ domains, 40+ services) and a proprietary industrial Enterprise Intent Corpus (EIC) collected from a real task-oriented trip planning system. For synthetic training, the authors generate 3,000 dialogues (36K turns) for SGD and 11K dialogues (64K turns) for EIC, removing low-quality turns via a filtering step.

The core architecture involves prompting a large language model (LLM) to generate multi-turn dialogue snippets conditioned on sequences of intents and attribute tuples. The attributes fall into two groups: topic attributes (class-dependent and independent metadata like location, request type) and style attributes (user linguistic behaviors such as formality, sentence structure, keyword use). Attributes are generated by zero-shot prompting the LLM to propose plausible values, then manually curated.

The generation function takes an intent ci along with topic and style attribute inputs to generate a chunk of 1-5 user-system turns maintaining the intent. For filtering, a separate LLM model plays the role of an intent judge, predicting the intended intent of each user utterance; samples with mismatched predicted intents are discarded.

For style diversification, two post-hoc stylization models are introduced: (1) Univ, which maps synthetic utterances to a single universal human-like style using a T5 or Llama backbone trained on paired synthetic-human utterance data; (2) Exam, which incorporates example utterances of the target style in-context for more fine-grained style adaptation.

These stylization models are trained on 23K/5K/4.6K train/test/val splits for EIC and 40K/9K/8.7K splits for SGD, using synthetic utterance/human reference pairs obtained via an LLM paraphrasing prompt. The stylization mapping transforms the LLM-generated utterance conditioned on previous system utterance and optionally example utterances.

For evaluation, intent classification accuracy and Macro F1 scores are used. Two standard classifiers, Llama-3.2-1B and DistilRoBERTa-base, are trained on the synthetic datasets and compared to human data upper bounds. Filtering impact, attribute ablations (no attributes, topic-only, style-only), stylization model ablations, and transfer to human and mixed data scenarios are analyzed. Linguistic metrics like lexical diversity, entropy, and readability measure stylistic differences between datasets.

Reproducibility: The authors indicate code release upon acceptance but datasets (especially EIC) are proprietary, limiting full reproduction. Model details and hyperparameters are given in an appendix. The method leverages publicly available LLMs and standard classifiers.

One example workflow involves prompting an LLM with an intent sequence plus specified style-topic attributes to generate a multi-turn dialogue chunk, filtering it with an intent-judging LLM, optionally applying post-hoc stylization with Univ or Exam, and then using the resulting synthetic utterances as training data for intent classifiers evaluated on held-out test utterances.

Technical innovations

• Explicit decomposition and use of style versus topic attributes for zero-shot, annotation-free synthetic dialogue generation, showing style diversity is more critical for data utility.
• Two novel post-hoc stylization models (Univ and Exam) to transform LLM outputs into diverse human-like styles by learning mappings from synthetic utterances to human references with or without example-based conditioning.
• Use of an LLM as an automated intent-judge filter to remove low-quality or off-intent generated utterances without requiring any annotation.
• Generation of multi-turn dialogue chunks conditioned simultaneously on intent sequences and style-topic attributes in a single LLM call to improve contextual coherence and reduce computational overhead.

Datasets

• Schema-Guided Dialogue (SGD) — 3,000 synthetic dialogues, 36,188 turns generated by this work; original dataset public (Rastogi et al., 2020).
• Enterprise Intent Corpus (EIC) — 11,017 synthetic dialogues, 63,945 turns; proprietary real-world industrial dataset from a trip planning dialogue system.
• Stylization datasets for Univ and Exam — 23,011 train, 4,826 test, 4,611 val samples (EIC); 39,931 train, 9,261 test, 8,771 val (SGD) from generated utterance pairs.

Baselines vs proposed

• Human annotated data: EIC accuracy (Llama IC) = 0.843; synthetic = 0.765 (90.7% of human)
• Human annotated data: SGD accuracy (Llama IC) = 0.875; synthetic = 0.817 (93.3% of human)
• No-Attribute baseline (Llama, EIC) F1 = 0.706 vs Ours (Style+Topic) = 0.735
• Topic-only baseline (Llama, EIC) F1 = 0.743 vs Style-only (Ours) = 0.763
• Univ stylization (T5 backbone) improves Llama IC on EIC: from 0.706 to 0.702 F1 (without filtering), Exam stylization improves to 0.684 (Table 6)
• Filtering with GPT-4-based judge boosts EIC F1 by ~2% for No-Attribute (0.706 to 0.725) and similar gains for other models (Table 4)
• DistilRoBERTa classifier benefits less from style attributes and stylization than Llama; style attributes degrade topic-only baseline for DistilRoBERTa on EIC.

Limitations

• Relies on large LLMs for generation and filtering which can be expensive and opaque; impact of cheaper/smaller models unclear.
• Evaluation limited to intent classification on two dialogue datasets; generalization to other dialogue tasks or domains is untested.
• Stylization step sometimes alters user intent (~7.4% error reported), which could be problematic depending on application tolerance.
• Dataset proprietary for EIC, limiting external reproducibility and benchmarking compared to fully public datasets.
• No adversarial testing against malicious or out-of-distribution inputs presented; robustness unassessed.
• Filtering depends on an LLM judge without human verification, possibly introducing subtle biases or errors.

Open questions / follow-ons

• How do different LLM sizes or architectures affect the quality and style diversity of synthetic dialogues?
• Can the stylization methods be generalized to more complex multi-turn dialogue coherence tasks beyond single-turn utterance style?
• How robust is the approach to adversarial or noisy input simulation, and does style diversity help against adversarial attacks?
• What are the trade-offs in utility and computational costs between incorporating style during generation versus post-hoc stylization at larger scale?

Why it matters for bot defense

This work is highly relevant for bot-defense engineers employing synthetic data for training NLP models in dialogue or intent classification tasks, particularly when labeled data is scarce or unavailable. The findings clearly show that controlling and diversifying linguistic style in synthetic utterances is crucial to avoid models exploiting superficial stylistic shortcuts instead of learning true semantic intent distinctions. The LLM-based filtering approach provides a scalable method to maintain data quality without manual annotation. For captcha or bot-detection systems relying on intent or NLP components, incorporating style diversity during synthetic data generation could improve robustness and generalization in adversarial, multilingual, or noisy environments. The post-hoc stylization models also suggest possibilities for adapting generic synthetic utterances into more human-nuanced styles of specific user populations commonly targeted in bot detection challenges. However, practitioners should note the potential risk of stylization altering semantic intent, which may adversely impact model accuracy if unchecked. Overall, the study advocates for far more nuanced treatment of linguistic style as an essential attribute rather than a minor nuisance in synthetic training data pipelines relevant to security and bot-defense.

Cite

@article{arxiv2606_20400,
  title={ The Significance of Style Diversity in Annotation-Free Synthetic Data Generation },
  author={ Zahra Abbasiantaeb and Zeno Belligoli and Omar Essam and Mohammad Aliannejadi },
  journal={arXiv preprint arXiv:2606.20400},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20400}
}

Read the full paper

• arXiv:2606.20400 (abstract)
• arXiv:2606.20400 (PDF)