SocietyBench: Forecasting Counterfactual Social-World Evolution
Source: arXiv:2608.04009 · Published 2026-08-04 · By Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi
TL;DR
SocietyBench addresses an underexplored capability of large language models (LLMs): forecasting the unfolding of real social events rather than merely completing specific tasks. Existing benchmarks focus heavily on task completion or binary event forecasting but fail to measure how well models understand and predict social dynamics holistically, especially public opinion shifts intertwined with factual developments. SocietyBench introduces a novel counterfactual social-world benchmark, anonymizing real event timelines by replacing entities and shifting dates to prevent memorization-based shortcuts. It evaluates models on two orthogonal axes—probability calibration of event occurrence and temporal accuracy of event timing—across five heterogeneous social events, producing a multi-dimensional leaderboard that reveals significant gaps remain. Leading LLMs achieve at best around 75/100, just halfway between random guessing (50) and perfect forecasting (100), and agent frameworks fail to improve upon base models. The wide per-event performance variance underscores the importance of multi-event evaluation. SocietyBench releases anonymized timelines, extensive question banks, ground truths, and scoring code to fuel further research.
The benchmark's innovation lies in its methodology combining a realistic, multi-platform data source, fact-plus-opinion temporal timelines, and a rigorous anonymization protocol to create counterfactual forecasting challenges. The design rewards uncertainty-aware prediction and nuanced temporal reasoning beyond binary yes/no calls, addressing gaps in prior forecasting and simulation benchmarks. Empirical evaluation of six leading LLMs, three agent systems, and two model-free heuristics demonstrates the task's difficulty and the weaknesses of current approaches. Notably, common agents based on plan-and-solve or debate fail to outperform their underlying LLM cores, and simple heuristics lag behind all LLMs. This work opens new directions for social intelligence evaluation in LLMs emphasizing genuine understanding and forecasting rather than recall or task execution.
Key findings
- On 5 diverse events and 125 prediction points in Chinese and English, the best LLM (GPT-5.5) scores 75.0/100 overall, only halfway between trivial anchor of 50 and perfect 100.
- The two evaluation axes—probability calibration and temporal accuracy—are orthogonal; models can be strong on one and weak on the other (e.g., Gemini-3.5-Flash good calibration but weaker timing).
- Per-event gaps reach up to 21.4 points on a single axis between best and worst models, underscoring unreliable conclusions from single-event evaluations.
- Agent frameworks (LangGraph, AutoGen, MiroFish) built on a common base LLM fail to improve calibration or timing scores, sometimes reducing performance by several points.
- Non-LLM baseline heuristics (frequency and momentum) score marginally above random guess (about 54 vs anchor 50) and far below all LLMs on calibration; they emit no temporal predictions.
- Anonymization via entity substitution and date shifting (up to ±180 days) effectively prevents memorization from pretraining, verified by adversarial LLM auditors failing to identify original events.
- Models are better at correctly rejecting false events (mean calibration 76.1) than confirming true ones (60.3), indicating a bias toward skepticism in forecasting.
- Later cutoff points (with more historical context) see modestly better calibration scores, validating the benchmark's temporal reasoning demands.
Threat model
The adversary is a trained language model or multi-agent system with access to large-scale pretraining corpora but without access to the anonymization keys that link replaced named entities back to original names or the true absolute calendar dates. The adversary cannot exploit memorized event identities or exact dates due to entity substitution and uniform date shifting, nor can it query external live data beyond limited internal resources. The model must rely on structural temporal patterns and social-dynamic reasoning rather than recall or lookup of known timelines.
Methodology — deep read
The task is formulated as a retrospective forecast at multiple cutoff dates along anonymized event timelines derived from real-world incidents. Each timeline combines factual news data and social-media public opinion signals from five platforms, producing a date-indexed sequence with two distinct layers: factual events and sentiment trends.
Threat Model: The adversary is the LLM or agent that has extensive access to pretraining data but must forecast the social world's future evolution from a given anonymized timeline ending at cutoff date d_P. Entities and absolute dates are obfuscated to prevent simple recall or lookup, effectively creating a counterfactual social world structurally identical to the real event but with different identifiers and time offsets. The adversary cannot identify or match events based on memorized surface features.
Data: Five heterogeneous social events are selected, covering public controversy, geopolitics, tech policy, financial markets, and trade policy. Each event's timeline contains 39 to 168 nodes (time points) and 25 prediction cutoffs, producing over 25,000 calibration questions and 3,100 temporal events total across both Chinese and English editions (each question bank is one-to-one bilingual). The events include: Wuhan Library harassment, US-Iran conflict, TikTok divestiture ruling, Super Micro NASDAQ crisis (SMCI), and Trump tariff escalation.
Anonymization: A three-phase pipeline replaces named entities via a per-event lookup table, shifts all dates by a uniform random offset sampled from U[-180d,+180d], and performs semantic consistency checks. An adversarial panel of three independent LLM auditors attempts to guess the original event and date window, triggering iterative augmentation or rejection of the timeline if identification confidence is high, ensuring memorization-based recall is minimized.
Architecture & Model Usage: The benchmark tests six API-accessible frontier LLMs (including GPT-5.5, Gemini-3.5-Flash, Claude-Opus-4.8, DeepSeek-V4-Pro, Kimi-K2.5, Doubao-Seed-2.0-Pro) and three agent frameworks built atop Doubao (LangGraph, AutoGen, MiroFish). Models are prompted deterministically (temperature=0) with full visible context up to cutoff d_P, then asked standardized calibration and temporal questions. External web search is disabled except within the agent frameworks as internally allowed.
Training: Not applicable, as evaluation is zero-shot on anonymized timelines. No fine-tuning is performed.
Evaluation Protocol: The benchmark provides two orthogonal scoring axes each normalized to 0-100. Calibration questions request a probability estimate p_q that an event E_q occurs within a date window W_q following cutoff d_q. Scores are weighted by question window length and distance from cutoff to emphasize near-future predictions and non-trivial windows. Weighted mean absolute error from true binary outcomes y_q forms the calibration score.
Temporal questions require predicting the exact date d_e of a future event e. Error is measured in days, normalized within contiguous 30-day buckets. The temporal score penalizes date discrepancies proportionally with proximity to the event date. These two scores are computed independently per event and then averaged across the five events for a headline metric.
Baselines: Two strong model-free heuristics based on frequency and momentum of past event rates are provided, supplying calibration probabilities but no temporal predictions.
Ablations: Various question subsets, scoring formulas, true/false question balance, and cutoff timing are ablated to validate robustness. The scoring is insensitive to these variations, and model ranking remains stable.
Overall, the pipeline creates a rigorous, multi-lingual, counterfactual forecasting benchmark with granular calibration and timing evaluation. Entropy in forecast difficulty arises naturally from event complexity and temporal horizons.
Example: At a cutoff during the Trump tariff escalation, a model must estimate the likelihood and timing of subsequent tariff announcements across a 90-day horizon without recognizing real entity labels or dates. It outputs probabilities for if/when events happen in defined windows, which are compared to ground truth from the shifted timeline, yielding a probability calibration score and a temporal error penalty. Models exhibiting overconfidence or freezing on heuristics perform poorly, while the leading LLMs partially but not fully capture evolving dynamics.
Technical innovations
- A counterfactual social-world anonymization pipeline combining per-event named-entity substitution and uniform date shifting (±180 days) to prevent memorization from pretraining data.
- A dual-layer timeline construction integrating factual events and public-opinion sentiment from multiple web and social-media platforms to capture complex social dynamics.
- A two-axis evaluation metric decomposing forecasting into orthogonal probability calibration and temporal accuracy components, each scored on calibrated 0–100 scales.
- A multi-event, bilingual (Chinese and English) end-to-end benchmarking pipeline with fully audited prediction banks and semantic consistency verification enabling reproducibility without expert labeling.
Datasets
- Wuhan Library — 39 timeline nodes, 2,862 calibration questions, 534 temporal events — public controversy event
- US-Iran Conflict — 41 nodes, 4,574 calibration questions, 503 temporal events — geopolitical conflict
- TikTok Divestiture — 127 nodes, 6,697 calibration questions, 571 temporal events — technology policy
- SMCI (Super Micro) Delisting — 40 nodes, 3,196 calibration questions, 528 temporal events — financial market crisis
- Trump Tariff Escalation —168 nodes, 8,035 calibration questions, 976 temporal events — trade policy
Baselines vs proposed
- Frequency baseline (calibration probability only): calibration score = 53.8 vs GPT-5.5 calibration 75.3
- Momentum baseline (calibration probability only): calibration score = 54.2 vs GPT-5.5 calibration 75.3
- Doubao-Seed-2.0-Pro LLM base: 64.3 calibration / 62.6 temporal vs GPT-5.5 LLM: 75.3 calibration / 74.6 temporal
- Agent LangGraph (on Doubao base): 62.0 calibration / 62.3 temporal vs Doubao base 64.3 / 62.6
- Agent AutoGen (on Doubao base): 58.5 calibration / 62.5 temporal vs Doubao base 64.3 / 62.6
- Agent MiroFish (on Doubao base): 63.1 calibration / 62.4 temporal vs Doubao base 64.3 / 62.6
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.04009.
Fig 1: | Why SocietyBench evaluates counterfactual forecasting rather than event recall. Evaluated
Fig 2: | SocietyBench methodology. Each retrospective cutoff 𝑑𝑃splits the anonymized timeline into a
Fig 3 (page 2).
Fig 4 (page 2).
Fig 5 (page 2).
Fig 6 (page 2).
Fig 7 (page 2).
Fig 8 (page 2).
Limitations
- The event set is limited to five social events, which may not cover the full diversity of social or geopolitical dynamics.
- No adversarial attacks or stress tests against deliberate model manipulation or deceptive inputs were conducted.
- The anonymization, while rigorously audited, may incompletely remove all forms of memorized knowledge leakage.
- Temporal granularity is limited to day-level buckets with a maximum prediction horizon of 90 days post-cutoff.
- Agent frameworks evaluated built on a single LLM base may not represent the full range of multi-agent or multimodal forecasting architectures.
- Per-question model responses are not released, limiting fine-grained error analysis by the community.
Open questions / follow-ons
- How can stronger anonymization techniques or synthetic counterfactual event generation further reduce memorization leakage?
- Can multi-agent or multimodal frameworks incorporating real-time social data improve forecasting beyond static LLMs?
- What are effective methods to incorporate finer-grained sentiment dynamics and complex public opinion shifts into social event forecasting?
- How would model performance scale with longer forecasting horizons or more diverse event types across global contexts?
Why it matters for bot defense
SocietyBench provides a rigorous evaluation framework to test the ability of large language models and agent systems to understand and forecast complex social dynamics rather than just perform scripted task completions. For bot-defense and CAPTCHA practitioners, this benchmark highlights the current limitations of frontier LLMs in modeling realistic social event trajectories, especially under counterfactual anonymization conditions. This indicates that directly leveraging LLMs for anticipatory social or adversarial prediction tasks remains challenging and requires cautious validation.
Additionally, the orthogonal calibration and timing scoring offers a nuanced lens on predictive uncertainty—useful for designing bot-detection systems that rely on forecasting human-like behavioral patterns evolving over time. The anonymization techniques also point to best practices for assembling evaluation datasets that prevent leakage and memorization, ensuring true generalization when simulating adversarial social interactions or dialogue-based CAPTCHAs. Finally, the persistent performance gaps and agent difficulties caution against overreliance on current reasoning wrapper techniques without deeper architectural innovation.
Cite
@article{arxiv2608_04009,
title={ SocietyBench: Forecasting Counterfactual Social-World Evolution },
author={ Zhenran Wang and Zhonghan Bian and Jinsong Li and Zhangyang Qi },
journal={arXiv preprint arXiv:2608.04009},
year={ 2026 },
url={https://arxiv.org/abs/2608.04009}
}