Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating

Source: arXiv:2607.28550 · Published 2026-07-30 · By Oscar Heath, Rohan Alexander

TL;DR

This paper addresses the problem of mode collapse in silicon sampling, where large language models (LLMs) generate synthetic survey responses with unrealistically low variance despite matching mean values well. The authors argue this occurs because LLMs struggle to generate numeric data that preserves the distribution shape and propose using Semantic Similarity Rating (SSR), a method that obtains text responses from LLMs and maps them to numeric scales via text embeddings. They systematically compare direct numeric prompting to SSR on thermometer-scale political attitude questions using four LLMs and ANES survey data from 2016 and 2020. Results show SSR substantially reduces mode collapse, improving fidelity of the synthetic distributions as measured by lower KL divergence to real data and better matching variances, without harming the accuracy of mean estimates. Moreover, the SSR approach only requires calibration of a single global temperature parameter, which generalizes well across years. However, SSR does not resolve systematic biases towards extreme responses found in LLM-generated data. The paper concludes SSR is a simple and effective technique to enhance silicon sampling realism for survey research, illustrating the benefits of leveraging LLMs’ strength in natural language generation combined with semantic embeddings for numeric scaling.

Key findings

• Using Semantic Similarity Rating (SSR) reduces KL divergence between synthetic and real ANES response distributions by over an order of magnitude compared to direct numeric prompting—for example, claude-sonnet-5 raw numeric KL=1.97 vs SSR KL=0.07 on 2016 data.
• SSR synthetic response distributions match real data variance substantially better: standard deviation of synthetic responses correlates closely with real ANES SD, whereas raw numeric outputs suffer from mode collapse with low variance (Fig 6).
• Mean absolute error of synthetic means remains mostly unchanged or slightly increases with SSR compared to raw numeric prompting, indicating SSR improves distribution shape without sacrificing mean accuracy (Fig 5).
• A single global temperature parameter T=0.25 calibrated on 2016 ANES data effectively controls variance in SSR distributions and generalizes well to 2020 ANES data, showing stable performance without overfitting (Table 1).
• Across all respondent-target group pairs (36 cells), SSR consistently outperforms numeric prompting in KL divergence across three LLMs: claude-sonnet-5, deepseek-v4-flash, and gpt-5.4-mini.
• LLMs tend to produce systematically biased extreme responses (more polarized opinions) which SSR does not correct, indicating remaining fundamental modeling limitations.
• Prompt changes to solicit text responses rather than numeric answers reduce mode collapse, leveraging LLMs’ natural language strengths.
• The approach is training-free and requires only a small calibration set of real data to tune the global temperature parameter.

Threat model

The adversary is a researcher or analyst using LLMs to generate synthetic survey responses via silicon sampling. The adversary has access to detailed persona-conditioned prompts but relies solely on the LLM's generative capacity without external randomization or stochastic calibration. The LLM is assumed unable to reliably produce numeric samples reflecting full real-world distribution variance, posing a risk of mode collapse and misleading inference. The adversary cannot modify LLM weights or access internal randomness controls; post-processing like SSR is feasible for improving sample fidelity.

Methodology — deep read

The authors study silicon sampling on political thermometer questions using ANES 2016 and 2020 data. They generate synthetic responses simulating actual respondents using frontier LLMs: claude-sonnet-5, deepseek-v4-flash, and gpt-5.4-mini.

1. Threat Model & Assumptions: Researchers assume an adversary that prompts an LLM with detailed persona descriptions but the LLM is only generating responses without additional external randomization or calibration. The key limitation is that LLMs cannot well generate numeric samples reflecting true distribution variance, leading to mode collapse.

2. Data: They use reproducibility datasets from Bisbee et al. (2024) containing ANES 2016 responses and persona prompts for 7530 respondents, each with four thermometer questions (Democratic Party, Republican Party, Liberals, Conservatives). For testing, they use 6630 personas from the ANES 2020 dataset processed similarly. Responses and demographic variables are used to construct persona prompts.

3. Architecture/Algorithm: Two silicon sampling methods are compared:

• Numeric prompting: LLMs are prompted to generate a direct numeric rating 0-100.
• Semantic Similarity Rating (SSR): LLMs generate free-text responses describing feelings toward groups. These texts are embedded using Gemini Embedding 2 to create vector representations. Five anchor points on the thermometer scale with corresponding anchor texts are also embedded. Cosine similarities between each response embedding and anchor embeddings produce similarity scores.

These scores are normalized (min-max) and passed through a softmax with a learnable temperature parameter T to convert similarities into a probability mass function over anchors. A Gaussian kernel density estimator then produces a smooth probability distribution over the 0-100 thermometer scale for each response. This converts text responses into calibrated numeric distributions.

4. Training Regime: Only the temperature parameter T (between 0 and 1) is trained by minimizing overall KL divergence between the synthetic (SSR) distributions and real ANES 2016 survey distributions, across all respondent-target pairs (36 combinations). T is searched in increments of 0.05. The optimum found is T=0.25. No other parameters or fine-tuning of the LLMs is done.

5. Evaluation Protocol: The main evaluation metric is the Kullback-Leibler (KL) divergence between real and synthetic score distributions, to measure distribution fidelity. They also assess mean absolute error (MAE) of synthetic vs real means for each group. Variance matching is analyzed by comparing synthetic and real standard deviations. Results are visualized for selected subpopulations by response distributions (density plots). The fitted temperature parameter from 2016 data is applied unchanged to 2020 to test generalization.

6. Reproducibility: Code and data for the study are available on Zenodo. The prompting templates from the prior study Bisbee et al. (2024) are reused. The ANES 2016 and 2020 public datasets support replication. The Gemini Embedding 2 embedding model is publicly referenced.

Concrete example: For one persona, the SSR pipeline first generates a text describing feelings toward say the Republican Party. The text is embedded and cosine similarities computed against anchor texts representing thermometer scores 0, 25, 50, 75, 100. The similarity vector is normalized, softmaxed using T=0.25, and results in a probability distribution over those scores, giving a synthetic response distribution rather than a single numeric point. Aggregating many such synthetic distributions yields a synthetic population response distribution closely matching the real ANES distribution in shape and spread.

Overall, the methodology clearly separates the failure mode—LLMs producing peaked numeric outputs—and introduces a novel SSR post-processing step to extract richer calibrated numeric distributions from free text responses. Results show SSR greatly reduces mode collapse and improves distributional fidelity without retraining or complex fine-tuning.

Technical innovations

• Application of Semantic Similarity Rating (SSR) to silicon sampling: generating text responses then mapping them to numeric distributions using embeddings rather than direct numeric prompting.
• Use of a single global temperature parameter in the softmax over cosine similarity scores to calibrate variance across all question groups without overfitting.
• Employing asymmetric embedding between response text (query) and anchors (documents) with Gemini Embedding 2 to improve similarity rating accuracy.
• Demonstration that SSR-derived synthetic distributions better match real survey variance and shape (KL divergence) than raw numeric output, supporting distributional fidelity improvements.

Datasets

• American National Election Studies (ANES) 2016 Time Series Study — 7530 personas — public dataset used for training and calibration
• American National Election Studies (ANES) 2020 Time Series Study — 6630 personas — public dataset used as out-of-sample test

Baselines vs proposed

• claude-sonnet-5 raw numeric: mean KL divergence = 1.97 vs SSR: 0.07 (2016 ANES)
• deepseek-v4-flash raw numeric: KL = 1.34 vs SSR: 0.11 (2016 ANES)
• gpt-5.4-mini raw numeric: KL = 0.61 vs SSR: 0.13 (2016 ANES)
• Mean absolute error of synthetic means is similar between raw numeric and SSR; SSR typically does not degrade mean accuracy
• KL divergence on 2020 ANES using SSR with 2016-calibrated T=0.25 remains low and comparable (e.g., claude-sonnet-5 SSR KL=0.08), indicating generalization

Limitations

• SSR does not correct systematic biases in LLM outputs such as an exaggerated tendency toward more extreme (polarized) responses.
• Calibration relies on availability of a real dataset for temperature parameter tuning; practicality may reduce if limited ground truth data is accessible.
• Only thermometer-scale survey questions were studied; applicability to other question formats or numeric scales remains unknown.
• Temperature parameter is global and not question- or subgroup-specific; finer calibrations might further improve performance but risk overfitting.
• LLMs still exhibit fundamental limitations in generating truly random numeric samples reflective of target distributions, which SSR can only partially mitigate.
• Evaluation focuses primarily on KL divergence and distributional shape; downstream impacts on survey analysis or policy inference require further study.

Open questions / follow-ons

• How to effectively debias LLM synthetic survey responses to correct systematic polarization or extremity biases beyond variance calibration?
• Can SSR or similar semantic embedding methods be generalized to non-thermometer survey scales or multi-choice categorical responses?
• What is the impact of temperature parameter calibration choices on downstream survey analysis, such as regression or subgroup difference estimation?
• Could fine-tuning or prompt engineering combined with SSR further reduce mode collapse or biases without requiring access to model weights?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners exploring the use of LLMs for behavior simulation or synthetic population generation, this paper offers an instructive case on a common failure mode: mode collapse due to LLM numeric response generation limitations. The Semantic Similarity Rating approach shows that generating text responses and then mapping them using text embeddings to numeric scales can yield synthetic data distributions that faithfully capture variance and shape of real data. This insight suggests that soliciting natural language outputs from LLMs, combined with embedding-based calibration, may be a generally useful technique to produce realistic synthetic behavioral distributions in tasks where numeric responses or scores are needed. Furthermore, the use of a simple temperature parameter to calibrate variance with minimal overhead is attractive for practical deployment without complex model retraining. However, the persistence of systematic biases in the synthetic data highlights the need for cautious interpretation and further bias mitigation, especially for security-critical applications where subtle distributional fidelity matters for adversarial robustness or detection. Overall, the paper’s methodology and findings provide a valuable blueprint for bot-defense engineers seeking to leverage LLMs as synthetic data generators while avoiding mode collapse pitfalls.

Cite

@article{arxiv2607_28550,
  title={ Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating },
  author={ Oscar Heath and Rohan Alexander },
  journal={arXiv preprint arXiv:2607.28550},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28550}
}

Read the full paper

• arXiv:2607.28550 (abstract)
• arXiv:2607.28550 (PDF)