Contextualized Early Detection of Online Firestorms: A Sequential LLM-Based Approach
Source: arXiv:2607.14957 · Published 2026-07-16 · By Besim Shala, Peter Mandl, Andreas Humpe, Martin Häusl
TL;DR
This paper tackles the challenge of early detection of online firestorms—rapid, escalating waves of highly negative user content on social media that can cause serious reputational harm. Traditional detection methods rely mainly on volume counts, sentiment scores, or predefined linguistic features that only indirectly capture evolving contextual meaning in discussion threads. The authors propose a novel Large Language Model (LLM)-based detection system with two modes: a Global Recognition Mode that retrospectively classifies entire Reddit threads by aggregating chunk-level LLM assessments, and a Sequential Early Warning Mode that processes threads as they evolve and issues alerts when calibrated thresholds across three indicators (negativity share, escalation level, contributor count) are met. Using a balanced Reddit dataset of 200 threads, the global mode achieved strong classification accuracy (91.5%) and the early warning mode achieved very high recall (98%) while detecting escalating threads after only about 4.8% of a thread had unfolded (average 8.56 comments and 4 distinct contributors). This work demonstrates that LLMs can effectively model discourse contextualization and meaning shifts to enable reliable early firestorm warnings, moving beyond static sentiment or volume-based heuristics.
Key findings
- Global Recognition Mode achieved overall accuracy of 91.5% on balanced Reddit dataset (100 firestorm, 100 non-firestorm threads).
- Firestorm class precision = 0.95 and recall = 0.88 in global mode, indicating conservative but accurate labeling.
- Early Warning Mode reached firestorm recall of 0.98, detecting 59 of 60 escalating threads.
- Early Warning Mode firestorm precision = 0.82 and non-firestorm precision = 0.98, reflecting deliberate sensitivity bias.
- False positive rate (FPR) in early warning mode was ~22% (13 false alarms out of 60 non-firestorm threads).
- Detection in early warning mode occurred on average after 8.56 comments and 4.02 distinct contributors (approx. 4.8% of thread progression).
- Chunk-level local assessments followed by hierarchical merging enabled long-context awareness that improved performance beyond comment-level sentiment approaches.
- Calibration via grid search over negativity share, escalation level, and contributor count thresholds enabled tunable trade-offs between recall, false positives, and detection timeliness.
Threat model
n/a — The study does not explicitly model an adversary or threat actor attempting to evade detection. The focus is on algorithmic detection of organically evolving online firestorms using social media discourse signals without assumptions about adversarial manipulation or capabilities.
Methodology — deep read
The study frames online firestorm detection as a sequence classification and early warning problem, operating on Reddit discussion threads labeled as firestorm or non-firestorm, balancing dataset size across classes. The adversary is not explicitly modeled because the focus is on algorithmic detection of organic discourse escalation. Data consisted of 200 publicly accessible Reddit threads evenly split into 100 firestorm and 100 non-firestorm cases, sampled across diverse thematic subreddits to avoid domain bias. Each thread retains comment order and contributor identities, with average firestorm threads showing nearly twice as many comments and contributors compared to non-firestorms, but with shorter comments on average.
The system has two main modes. Global Recognition Mode segments complete threads into chronological chunks up to 12,000 characters, preserving local discourse coherence. Each chunk is independently scored by an LLM (OpenAI GPT-4o mini) prompted with firestorm definitions to produce a binary firestorm/no-firestorm label and a summary. A second LLM step hierarchically merges chunk summaries and labels to make a global thread-level prediction with justification, leveraging hierarchical context merging to scale to long inputs.
Early Warning Mode processes threads incrementally using sliding windows over comments (steps of 5 during runtime, max horizon 100 comments). A calibration phase uses grid search over 125 threshold combinations for three LLM-estimated indicators—negativity share (fraction of negative comments), escalation level (degree of piling-on and intensification), and contributor count (number of distinct conflict amplifying users)—based on 80 calibration threads (40 firestorm, 40 non-firestorm). The scoring function prioritizes firestorm recall with penalties for false positives and delayed detection.
During runtime, each window is fed to the LLM to extract the three indicators plus a categorical thread state (emerging/full firestorm). If all thresholds are met, a warning is triggered immediately; otherwise, the window slides until max horizon or thread end. The system enforces structured JSON outputs via schema constraints to automate parsing.
Evaluation uses thread-level binary ground truth labels. For global mode all 200 threads are used, computing accuracy, precision, recall, and F1. For early warning, 120 holdout threads (60 firestorm, 60 non-firestorm) are tested to ensure strict separation from calibration. Metrics include precision, recall, F1, false positive rate, mean comments until detection, and mean contributing users until detection.
Hardware and exact training hyperparameters are unreported since no fine-tuning is involved; the model operates solely via prompt-based evaluation calls. The study analyzes one concrete example of a Reddit thread segmented into chunks, scored locally, and then aggregated globally, as well as sequential sliding window evaluation producing early warnings within under 10 comments. Code or pretrained weights are not released, restricting exact reproducibility but input prompt formulas and threshold values are fully disclosed.
Technical innovations
- Extension of the LLM-as-a-Judge paradigm from static, single-shot evaluation to dynamic, sequential monitoring of evolving social media discussion threads.
- Hierarchical chunk-based input segmentation with local LLM assessments followed by LLM-based global merging enables effective long-context firestorm classification.
- Design and use of three theoretically grounded firestorm escalation indicators—negativity share, escalation level, contributor count—estimated continuously via LLM prompts in sliding windows.
- Calibration of a multi-threshold, rule-based warning decision function over LLM outputs to achieve tunable trade-offs favoring early detection recall at acceptable false positive rates.
Datasets
- Reddit Firestorm Dataset — 200 threads (100 firestorm, 100 non-firestorm) — public Reddit discussions across diverse subreddits (politics, consumer, entertainment, news, science, meta, gaming).
Baselines vs proposed
- Traditional sentiment analysis firestorm detection: 70-75% accuracy [prior literature] vs proposed Global Recognition Mode: 91.5% accuracy.
- Early warning systems with volume or linguistic feature thresholds typically do not report precise recall/FPR; proposed LLM sequential mode achieves firestorm recall 0.98 with FPR approx. 0.22.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14957.
Fig 1: Global Recognition Mode pipeline with local LLM-based chunk
Fig 2: Two-stage Early Warning Mode with threshold calibration (Stage 1)
Limitations
- Dataset limited to Reddit, which has unique discourse norms and platform features; generalizability to other platforms (Twitter/X, Facebook) is untested.
- Single annotator labeled threads, so systematic labeling bias cannot be ruled out without inter-annotator agreement metrics.
- Exact firestorm escalation onset points were deliberately not annotated due to ambiguity, limiting fine-grained temporal validation.
- Model choice constrained to GPT-4o mini; potential impact of model selection or fine-tuning remains unexplored.
- No adversarial evaluation or robustness tests against manipulated content or coordinated campaigns.
- Limited availability of code or pretrained weights restricts exact reproducibility and independent validation.
Open questions / follow-ons
- How well does the LLM-based sequential detection approach generalize to other platforms with different interaction structures, such as Twitter or Facebook?
- Can multi-annotator labeling protocols improve robustness and reduce bias in firestorm ground-truth assignments?
- How resilient is the detector to adversarially crafted text designed to evade LLM-based judgment or mimic non-escalating discourse?
- What improvements or trade-offs arise from integrating fine-tuning or domain-adaptive training of LLMs on firestorm data?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work illustrates the powerful contextual capabilities of LLMs beyond keyword or volume heuristics to dynamically monitor evolving discourse for early signs of coordinated negative escalation. The multi-indicator approach (negativity, escalation intensity, contributor count) shows how nuanced semantic and structural features can be synthesized by LLMs into actionable early warnings. Incorporating such sequential context-aware evaluation into automated moderation or risk detection pipelines could improve sensitivity to rapidly changing threat environments like coordinated disinformation or harassment campaigns. However, the moderate false positive rates and calibration steps highlight the operational challenge of balancing prompt detection with alert fatigue, a critical consideration for real-time bot or abuse detection systems. Overall, it suggests LLMs are promising as sequential evaluators of evolving conversational context, a perspective potentially applicable to CAPTCHA challenge design or bot behavior modeling.
Cite
@article{arxiv2607_14957,
title={ Contextualized Early Detection of Online Firestorms: A Sequential LLM-Based Approach },
author={ Besim Shala and Peter Mandl and Andreas Humpe and Martin Häusl },
journal={arXiv preprint arXiv:2607.14957},
year={ 2026 },
url={https://arxiv.org/abs/2607.14957}
}