Will Scaling Improve Social Simulation with LLMs?

Source: arXiv:2607.02464 · Published 2026-07-02 · By Caleb Ziems, William Held, Su Doga Karaca, David Grusky, Tatsunori Hashimoto, Diyi Yang

TL;DR

This paper investigates whether scaling large language models (LLMs) improves their fidelity in social simulation tasks, a promising but currently underdeveloped research area. The authors study three social simulation sub-domains—opinion modeling, behavioral simulation, and longitudinal forecasting—using a suite of 85 transformer models (Qwen3 architecture) trained on a fixed web corpus across compute budgets from 10^18 to 10^20 FLOPs. They then evaluate 35 larger open-weight models up to 70B parameters to predict downstream accuracy from loss. The key finding is that many opinion and behavior simulation tasks exhibit strong log-linear compute scaling, implying that simulation fidelity will improve substantially with increased scale, especially for populations well-represented in English web data. However, longitudinal forecasting and behaviors relying on human cognitive biases or complex reward learning scale more slowly or not at all. Fine-tuning on some null scaling tasks also failed to improve performance. This nuanced conclusion highlights that scaling helps social simulation broadly but outliers remain, especially in low-resource or out-of-distribution settings.

Key findings

• Compute scaling laws explain 85%-97% of variance in loss for opinion modeling (WVS, r2=0.85), behavioral simulation (Psych-101, r2=0.97), and longitudinal forecasting (ACL, r2=0.88) when scaling from 10^18 to 10^20 FLOPs.
• Observational scaling laws on 35 open-weight models (0.5B to 70B params) link loss to accuracy, allowing downstream performance predictions.
• Opinion and behavior simulation tasks where populations are well-represented in English web corpora show strong correlation with general capabilities (up to r2=0.60 for Canada in WVS), indicating scale will improve fidelity.
• Underrepresented populations (e.g., Pakistan r2=0.06, Myanmar r2=0.08) have weak correlations, attributed to sparse pre-training data representation as supported by Spearman rho=0.8 between corpus frequency and scaling fit.
• Longitudinal forecasting tasks improve slowly with scale (r2=0.60) and require over 1000× more compute to progress 5% in accuracy; estimated max accuracy ~77% with current scaling trends.
• About one-third of behavioral simulation subtasks show negligible or no scaling effects (r2<0.3) with some tasks having zero slope in loss-accuracy fits, particularly those involving human cognitive biases like risk aversion and correlated rewards.
• Fine-tuning Llama3 and Qwen2.5 models from 0.5B to 8B parameters on null scaling tasks failed to improve performance, reinforcing that some behaviors do not scale with model size or post-training.
• General capabilities related to knowledge and reasoning (e.g., MMLU) predict simulation fidelity better than programming or multi-step formal reasoning benchmarks.

Threat model

n/a — This is a foundational empirical study on the capabilities and scaling of LLMs in social simulation, not a security or adversarial robustness paper.

Methodology — deep read

The authors frame social simulation as a problem of modeling conditional probability distributions over finite outcome spaces using multiple choice question (MCQ) formats. They focus on three inference problems aligned to standard social science experimental designs: (1) Opinion simulation via World Values Survey (WVS) data modeling diverse demographic groups across 7 variables; (2) Behavioral simulation via Psych-101, a collection of psychological experiments including reward maximization and cognitive heuristics; and (3) Longitudinal forecasting via Americans' Changing Lives (ACL), predicting future social variables from past survey waves.

For compute scaling, they trained a controlled suite of 85 Qwen3 transformer LLMs with sizes 0.2B to 12B parameters on the DCLM web corpus, constrained to fixed compute budgets spanning 10^18 to 10^20 FLOPs. IsoFLOP methodology selects compute-optimal model/token ratios for sweeping scale. Model loss is measured on held-out test sets constructed to reliably evaluate opinion distributions (KL divergence), behavioral actions (cross-entropy), and longitudinal outcomes.

Following initial scaling law fits, the authors evaluate 35 open-weight base models from 7 families, ranging 0.5B to 70B parameters, each with varying architecture and training recipes. These models are evaluated across the three social simulation tasks and on a battery of 12 general capability benchmarks (knowledge, reasoning, instruction following, calibration, programming, etc.). Principal component analysis (PC1) reduces general benchmark performance to a single score.

They use observational scaling laws to correlate PC1 and individual general capabilities with social task performance, analyzing which capabilities best explain simulation fidelity variation. Linear and sigmoidal functions calibrate task loss to accuracy to predict future gains with scale. Correlations also relate simulation fidelity variation across demographics to distributional biases in pre-training data by counting corpus keyword frequencies.

In behavioral simulation, they further analyze subtask variation by psychological domain and finer experiment hypotheses, identifying a subset of tasks with negligible scaling tied to cognitive biases absent in LLMs. They fine-tune models on these 'null scaling' tasks to test whether supervised tuning improves performance, finding no scaling effect. All datasets, fine-tuned models, and evaluation codes are released for reproducibility.

Overall, the paper combines controlled isoFLOP scaling training, multi-model observational benchmarking, demographic data-driven analysis, and fine-tuning ablations to rigorously study how social simulation fidelity evolves with model scale and general capabilities.

Technical innovations

• Novel application of compute-optimal IsoFLOP scaling schedules to 85 LLMs for social simulation tasks on WVS, Psych-101, and ACL datasets.
• Use of observational scaling laws linking downstream social simulation accuracy to model loss, enabling extrapolations beyond current model sizes.
• Correlation analysis between pre-training corpus representation (frequency of location keywords) and social simulation scaling efficacy to quantify data distribution biases.
• Fine-grained subtask clustering within behavioral simulation to identify specific cognitive biases and learning phenomena that fail to scale with model size.
• Open release of fine-tuned Llama3 and Qwen2.5 models on social simulation tasks to enable community replication.

Datasets

• World Values Survey (WVS) 7th wave — varies by country demography — public survey data
• Psych-101 collection of psychological experiments — thousands of trials — from Binz et al. (2025)
• Americans' Changing Lives (ACL) longitudinal panel survey — 6 waves over 1986-2019 — public US social science dataset
• DCLM web text corpus — baseline pre-training data for Qwen3 models — public aggregate corpus

Baselines vs proposed

• Qwen3 suite (0.2B–12B params): task loss decreases log-linearly with compute, r2=0.97 for Psych-101, r2=0.85 WVS, r2=0.88 ACL
• Observed models (0.5B–70B params): downstream accuracy correlates with loss via sigmoidal/linear fits, e.g. Psych-101 max acc ~72% at 14B (Qwen2.5), requiring ~40× compute for 90%
• Fine-tuned Llama3 and Qwen2.5 (0.5B–8B params): no noticeable accuracy improvements on null scaling behavioral tasks

Limitations

• Compute scaling laws derived mostly on smaller controlled models (<12B parameters), with extrapolation needed for larger models, which may be unreliable.
• Observational scaling assumes that correlations between loss and accuracy continue beyond observed scale, which is uncertain.
• Datasets and tasks focus on MCQ-style finite discrete spaces; results do not necessarily generalize to open-ended or multi-agent simulations.
• Behavioral simulation domains with cognitive biases and complex correlated rewards empirically fail to scale, highlighting limitations in current LLM architectures or training.
• Underrepresentation in pre-training data limits performance and scaling efficacy for some demographic groups, reinforcing biases rather than eliminating them.
• Fine-tuning experiments are limited to up to 8B parameters and may not have tested all tuning or data augmentation strategies to rescue null scaling tasks.

Open questions / follow-ons

• Can targeted data augmentation or multimodal pre-training improve scaling behavior on underrepresented demographic groups and low-resource social domains?
• What architectural or training modifications enable LLMs to better capture human cognitive biases and heuristics in behavior simulation tasks?
• How do social simulation scaling laws extend to open-ended, multi-agent, and interactive simulations beyond finite MCQ settings?
• To what extent can fine-tuning or reinforcement learning from human feedback break the null scaling observed on correlated reward and risk aversion tasks?

Why it matters for bot defense

Bot-defense and CAPTCHA systems often rely on detecting human-like behaviors or decision patterns across social-psychological dimensions, which this work helps clarify regarding LLM simulation fidelity trends. The finding that scaling improves many social simulation tasks implies that larger LLMs could better mimic or predict human opinion distributions and typical behaviors, potentially challenging CAPTCHA designs reliant on surface behavioral cues. However, the areas where scaling fails—such as capturing human cognitive biases and complex reward learning—highlight enduring gaps in LLM human-likeness. Practitioners should note that LLM-based adversaries might become more socially sophisticated with scale in high-resource language contexts, but models remain imperfect in faithfully simulating nuanced human cognition and underrepresented groups. Defensive designs may benefit from exploiting these known weaknesses in bias calibration and longitudinal dependency handling, which do not improve with scale alone according to this study.

Cite

@article{arxiv2607_02464,
  title={ Will Scaling Improve Social Simulation with LLMs? },
  author={ Caleb Ziems and William Held and Su Doga Karaca and David Grusky and Tatsunori Hashimoto and Diyi Yang },
  journal={arXiv preprint arXiv:2607.02464},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02464}
}

Read the full paper

• arXiv:2607.02464 (abstract)
• arXiv:2607.02464 (PDF)