Online Safety Monitoring for LLMs

Source: arXiv:2607.02510 · Published 2026-07-02 · By Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, Eric Nalisnick

TL;DR

The paper addresses the problem that despite alignment training, large language models (LLMs) remain prone to generating unsafe outputs during deployment. To reduce harmful outputs, the authors focus on the crucial task of online safety monitoring, i.e., monitoring LLM outputs in real-time and raising alarms as soon as safety cannot be assumed. They study a simple monitor design that relies on thresholding a safety signal derived from an external verifier model, with calibration of the threshold carried out via risk control techniques offering probabilistic guarantees on false alarm rates.

Experimental evaluation on two safety domains—mathematical reasoning (factual correctness) and red teaming (toxicity/malicious use)—shows that this simple risk-controlling thresholding method performs competitively with more complex sequential hypothesis testing-based monitors (e-valuators). The simpler monitors detect unsafe outputs earlier with comparable or better false alarm control, an important property for limiting user exposure to harmful outputs. An additional experiment explores replacing expensive external verifier signals with cheaper internal token log-probabilities, showing a substantial drop in detection power, thus highlighting the need for informative signals.

Overall, the paper advances the state-of-the-art in practical, theoretically principled online LLM safety monitoring with low computational overhead and concrete statistical guarantees, while exposing key limitations and directions for future work.

Key findings

• Risk-controlling threshold monitors (CRC and UCB) effectively maintain false alarm rates below target levels (e.g., ϵ = 0.05, 0.1) on the MATH dataset, verified by confidence intervals over 10 runs (Fig. 1).
• CRC and UCB detect unsafe outputs earlier in generation (about halfway through output sequences) compared to e-valuator monitors which detect later despite higher power (Fig. 1, detection delay).
• On the Anthropic Red Teaming dataset, e-valuator-PAC achieves highest power (~80%) for harmful output detection, but CRC and UCB achieve comparable power on the FineHarm dataset where the verifier is fine-tuned for token-level harmfulness (Fig. 2).
• False alarm control is reliably maintained across datasets and monitors, despite variability in power based on verifier signal quality (Fig. 2).
• Replacing external verifier signals with generator internal token log-probabilities drastically reduces power from >90% to ~50% at similar false alarm levels, demonstrating token log-probs are a substantially weaker safety signal (Fig. 3).
• The simple single-threshold monitor requires no additional density estimators or temporally varying thresholds, leading to very low computational overhead.
• High-probability risk control (UCB) provides stronger safety guarantees than in-expectation control (CRC) but often at the cost of more conservative thresholds and lower power.
• Calibrated monitors can continuously inspect outputs token-by-token, raising alarms as soon as the safety signal falls below the threshold.

Threat model

The adversary is the natural generation process of the LLM which can produce unsafe (incorrect or harmful) outputs; they do not explicitly model an active attacker that manipulates or crafts adversarial inputs. The monitor has access only to an imperfect safety signal from an external verifier or internal statistics. It cannot access ground-truth safety labels at deployment and cannot fully prevent all unsafe outputs but aims to flag them early with controlled false alarm rates.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicitly the natural distribution of LLM outputs, which can produce unsafe sequences (incorrect, harmful). The monitor observes a proxy safety signal for each token or step output, such as the probability from an external verifier model. It cannot see true safety labels at deployment, only approximate signals. The monitor aims to raise alarms early upon detecting an unsafe state.

2. Data: Two main datasets are used: (i) MATH dataset for factual correctness in stepwise math problem solving; (ii) Anthropic Red Teaming and FineHarm datasets for multi-turn harmful/toxic dialog interactions. Calibration datasets with labels are drawn from the same distribution as deployment data. The MATH experiments use Claude Haiku 4.5 (90% correct) and Mistral-7B (26% correct) LLMs as generators, with external verifiers like Qwen2.5-Math-PRM-7B for signals. Labels y indicate safety (safe = 1, unsafe = 0).

3. Architecture / Algorithm: The core algorithm is a simple stopping rule monitor Φt = 1{∃k ≤ t: sk < λ} with a single scalar threshold λ applied uniformly at all time steps to safety signals st. An alarm is raised at the first step where the safety signal falls below λ. Calibration chooses λ to control either the false alarm risk RI(λ) = P(st < λ | safe sequence) or missed detection risk RII(λ).

Two risk control methods are used: (a) Conformal Risk Control (CRC) ensures expected false alarm rate ≤ ϵ by selecting λ with empirical risk plus finite-sample correction ≤ ϵ; (b) Upper Confidence Bound (UCB) control uses concentration inequalities (Hoeffding-Bentkus bound) to provide high-probability guarantees that false alarm risk ≤ ϵ with confidence 1−δ, producing more conservative thresholds.

4. Training Regime: No training of the monitor itself; calibration is performed on held-out labeled datasets by grid-searching λ over calibration sets for risk control. The verifiers are externally trained models (e.g. PRMs, Llama Guard), not trained jointly. Experiments use multiple seeds (10 runs) to assess variability.

5. Evaluation Protocol: Metrics include false alarm rate (false positives on safe sequences), power (true positive detection rate on unsafe sequences), and detection delay (fraction of sequence generated before alarm). Monitors compared include CRC, UCB, and e-valuator baselines that use sequential hypothesis testing with density estimators per step. Experiments cover both controlling false alarm risk and missed detection risk. Ablations test external verifier signals vs internal token log-probabilities. Statistical significance is assessed via confidence intervals and repeated runs.

6. Reproducibility: Code is publicly released at https://github.com/monasch/llm-monitor. Calibration datasets are standard published benchmarks (MATH, Anthropic Red Teaming). Verifier models are described with references. There are no closed datasets. Frozen verifiers used.

Example end-to-end: For a mathematical reasoning problem, the generating LLM outputs tokens stepwise. At each step t, a verifier model predicts the probability that the answer so far is correct (signal st). The monitor applies threshold λ chosen by CRC on calibration data to st continually. When st < λ, the monitor raises an alarm, halting generation or triggering review. The threshold λ is chosen so that the false alarm rate over calibration samples is ≤ ϵ, e.g., 5%. Evaluation tracks how often alarms correspond to truly incorrect answers (power) and how early the alarm is raised (detection delay).

Technical innovations

• Propose a simple online LLM safety monitor based on thresholding a verifier-derived safety signal, calibrated via conformal risk control with statistical guarantees.
• Demonstrate that a single, fixed threshold applied uniformly over the output stream yields competitive safety monitoring performance compared to complex sequential hypothesis testing approaches (e-valuators).
• Establish methods for calibrating thresholds both in expectation and with high-probability control guarantees using concentration bounds (Hoeffding-Bentkus).
• Empirically show that external verifier signals offer significantly better monitoring power than cheaper internal token log-probabilities, quantifying the cost-quality trade-off.

Datasets

• MATH — 12,500 math problems — public benchmark for mathematical reasoning
• Anthropic Red Teaming — ~10k multi-turn conversations — red team dataset with safety labels
• FineHarm — thousands of single-turn harmful interaction samples — compiled from WildGuard and WildJailbreak datasets

Baselines vs proposed

• E-valuator-anytime: controlled false alarm at ϵ=0.05 but violated bound on Mistral model; power higher than CRC but slower detection.
• E-valuator-PAC: highest power on Anthropic Red Teaming (~0.8) vs CRC/UCB power (~0.6–0.7); detects later than CRC/UCB.
• CRC: false alarm controlled and detects unsafe outputs earlier (~50% through sequence) than e-valuators; power slightly lower than best e-valuator.
• UCB: stronger false alarm control guarantee; more conservative thresholds; similar power and detection delay to CRC.
• On MATH with internal log-probability signals, power dropped from ~0.9 (external PRM) to ~0.5 at matched false alarm levels across monitors.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02510.

Fig 1: Monitoring factuality on mathematical reasoning

Fig 2: Monitoring harmlessness: All monitors maintain false

Fig 3: Signal ablation of token log-probabilities vs. external PRM: False alarm rate, power, and detection delay on Mistral-7B-Instruct

Fig 4: Monitoring performance when controlling missed detection risk RII (Eq. (7)) instead of false alarm risk RI (Eq. (2)): CRC and

Limitations

• The monitor’s performance is fundamentally limited by the informativeness and reliability of the safety signal; weak signals like token log-probs degrade performance.
• The single-threshold approach ignores temporal dependencies and varying difficulty across output steps, potentially missing opportunities for improved performance via per-step thresholds.
• No adversarial robustness evaluation was conducted; external verifiers could be vulnerable to adaptive attacks or manipulation.
• Calibration requires access to labeled held-out data drawn from the deployment distribution, which may not be feasible for all applications.
• The high-probability control method (UCB) is more conservative and requires more calibration samples, which might not scale well in some settings.
• Evaluation is limited to two safety scenarios (mathematical reasoning factuality and harmful output detection), but other safety dimensions may behave differently.

Open questions / follow-ons

• How to construct or learn more robust and informative safety signals that balance monitoring accuracy and computational cost?
• Can temporal structure be effectively incorporated via per-step or adaptive thresholds to improve detection power and reduce false alarms?
• What are the robustness properties of risk-controlling monitors under adversarial or distribution shift attacks on the verifier signals?
• How to extend the framework to jointly monitor for multiple safety dimensions (toxicity, factuality, bias) simultaneously in a scalable manner?

Why it matters for bot defense

For bot-defense and CAPTCHA systems leveraging or integrating LLMs, this paper provides a practical, low-overhead approach to flagging unsafe or manipulated LLM outputs in real-time to prevent abuse or harmful content generation. The approach’s reliance on a calibrated threshold over a safety signal ensures controlled false alarms, which is critical in user-facing contexts to avoid degrading user experience.

The findings about external verifier signals outperforming internal confidence scores guide defenders to prioritize model-agnostic but trustworthy safety checks rather than solely relying on the LLM’s own scoring mechanisms. Additionally, the demonstrated early detection capabilities reduce exposure to harmful content, informing deployment strategies where safety interruptions must be balanced with service fluency. However, the approach’s dependence on calibration data and verifier quality should caution practitioners to carefully design or vet their safety signals in production settings.

Cite

@article{arxiv2607_02510,
  title={ Online Safety Monitoring for LLMs },
  author={ Mona Schirmer and Metod Jazbec and Alexander Timans and Christian Naesseth and Maja Waldron and Eric Nalisnick },
  journal={arXiv preprint arXiv:2607.02510},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02510}
}

Read the full paper

• arXiv:2607.02510 (abstract)
• arXiv:2607.02510 (PDF)