Cybersecurity Detection Classification with Reasoning-enabled Language Models

Source: arXiv:2607.28460 · Published 2026-07-30 · By Amol Khanna, Manu Nandan, Cristian Viorel Popa, Joan Pujol-Roig, Diana Bolocan, Laura Vasilie et al.

TL;DR

This paper addresses the acute problem of alert fatigue in Security Operations Centers (SOCs) caused by an overwhelming volume of security detections that analysts cannot triage manually at scale. Prior approaches used large language models (LLMs) to directly classify alerts as malicious or benign but did not leverage the LLMs’ abilities to generate explicit reasoning for their decisions. The authors propose a novel triage system that trains a large LLM to generate chain-of-thought (CoT) reasoning traces before outputting a classification label, thereby making the model’s decision process transparent and evaluable. To compensate for the fact that CoT outputs degrade the reliability of label confidence estimates derived from model token probabilities, they additionally train a calibrator that reads the entire reasoning trace along with the label to estimate confidence in the verdict. The system is trained on a large, real-world dataset of human-labeled Windows endpoint detections using a four-stage method: automated prompt optimization, self-training with rationalization, reinforcement learning with verifiable rewards, and confidence calibrator training.

Empirically, the resulting model achieves 82.6% classification accuracy on held-out test data and significantly improves high-confidence automated triage: at thresholds favored by SOC analysts, benign recall improves by 43.0 percentage points and malicious recall by 18.3 points compared to a direct-label classifier baseline. The authors also show that the calibrator is essential, as an untrained confidence judge collapses high-confidence recall to zero, and that a domain-specialized 30B model trained with their method outperforms much larger general-purpose LLMs. This work establishes the value of combining explicit reasoning and confidence calibration for more effective automated threat triage in cybersecurity operations.

Key findings

• Final system reaches 82.6% test accuracy on real Windows endpoint detection classification.
• At analyst-preferred high-confidence thresholds, the system improves benign recall from 21.8% to 64.8% (+43.0 points) and malicious recall from 34.7% to 53.0% (+18.3 points) over a direct-label LLM baseline.
• Calibrator training is necessary: an untrained calibrator collapses high-confidence recall to zero on both benign and malicious classifications.
• Self-training alone improves validation accuracy by nearly 10 points to 81.7%, with continued gains from reinforcement learning to 87.7%.
• Reinforcement learning from verifiable rewards (GRPO) without self-training yields 61.6% benign recall but lower malicious recall than the combined approach.
• Domain-specialized Nemotron-3-Nano-30B outperforms larger general-purpose LLMs (e.g., Claude Opus 4.6) on classification accuracy and balanced precision/recall.
• High-precision benign recall drifts from 98.0% on validation to 90.8% on test data, revealing distribution shift sensitivity and motivating ongoing calibration in deployment.
• Training stages (prompt optimization, self-training, reinforcement learning) each provide additive improvements to calibrated high-confidence performance.

Threat model

The adversary manipulates or triggers cybersecurity detections that the system must classify as malicious or benign. The system assumes attackers cannot manipulate the model internals, training process, or the calibrator outputs. The adversary’s capabilities include generating detection events that may look malicious or benign to evade or trigger triage but cannot forge reasoning traces or undermine confidence calibration. The goal is reliable high-confidence automated triage, minimizing false positives and negatives under realistic attack conditions derived from operational SOC data.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicitly represented by malicious or benign Windows endpoint detections collected in a real SOC environment. The model’s task is to classify whether a detection is a true threat or false positive. The attacker is assumed not to control the model or training data but can influence detection distribution in deployment. The adversary cannot forge the reasoning trace or confidence calibrator outputs.

2. Data: The dataset consists of human-labeled Windows endpoint sensor detections collected over consecutive periods: 388,336 for training, 59,162 for validation, and 42,686 for testing. Each detection is a multi-field JSON record aggregating various sensor details (process lineage, prevalence, MITRE ATT&CK mappings, metadata, and textual context). Labels are binary: malicious or benign. No mention of data augmentation or additional preprocessing.

3. Architecture/Algorithm: The core classifier is a Nemotron-3-Nano-30B autoregressive LLM fine-tuned to produce a chain-of-thought (CoT) reasoning trace enclosed in <think>...</think> tags followed by a label token. A separate Nemotron-3-Nano-30B model serves as a confidence calibrator, trained to read inputs including the detection, CoT trace, and label to predict whether the classification is correct. Training includes standard supervised fine-tuning adapters (LoRA).

4. Training Regime:

• Prompt Optimization: Using the GEPA framework with a Nemotron-3-Super-120B reflection model for automated prompt mutation and scoring with an LLM-as-a-judge metric considering reasoning quality, coverage, and generalization, constrained to 1024 tokens.
• Self-Training: Extending AdaSTaR methods, the model iteratively samples and retains correct reasoning chains, rationalizes incorrect cases with LLM-generated explanations grounded in gold labels, and fine-tunes with LoRA adapters over 25 iterations. Consistency and accuracy monitored.
• Reinforcement Learning with Verifiable Rewards: Using GRPO algorithm, the model receives reward 1.0 only if output follows the required format and matches the ground-truth label, otherwise 0. Run for 8000 steps, with checkpoints selected by validation accuracy.
• Confidence Calibration: A separate calibrator is trained supervised on generated rollouts labeled as correct or incorrect. The calibrator outputs a binary correctness verdict, whose softmax provides a confidence score.

5. Evaluation Protocol: Metrics include accuracy, benign recall at 98% precision (high confidence), and malicious recall at 99% precision, reflecting automated triage operating points meaningful to human analysts. Medium and low confidence tiers at relaxed precision thresholds are also measured. Ablation studies remove or isolate training stages and the calibrator to understand contributions. Multiple baseline LLMs, including larger general-purpose models (e.g., Claude variants) are compared zero-shot. Evaluation includes test set of ~42,000 samples, with consistency measured over multiple stochastic forward passes.

6. Reproducibility: Code details and prompt examples are referenced to appendices but no public code or model weights are explicitly stated as released. Dataset is proprietary data from a collaborating SOC, thus closed. Training uses large GPU clusters (32 B200 GPUs) and standard Nvidia LLM infrastructure. Exact random seed and hyperparameter details, beyond broad iterative counts and resource settings, are not fully specified.

Example End-to-End: Given a JSON detection describing process lineage and metadata, the LLM classifier generates a detailed CoT reasoning trace assessing each field and their threat or benign implications (e.g., prevalence, parent processes), concluding with a benign or malicious label token. The calibrator reads the detection plus CoT and outputs a confidence estimate that this verdict is correct. High-confidence benign cases can then be automatically triaged/closed by SOC systems, saving analyst time.

Technical innovations

• Training an LLM triage classifier explicitly conditioned to output chain-of-thought reasoning traces before classification in cybersecurity detection classification.
• A novel four-stage training pipeline combining automated prompt optimization, self-training with rationalization, reinforcement learning from verifiable rewards, and dedicated confidence calibrator training for improved high-confidence triage.
• Use of a separate confidence calibrator model reading complete reasoning traces alongside labels to generate reliable confidence scores, overcoming overconfidence problems caused by CoT reasoning.
• Demonstration that calibrator training is essential - an untrained calibrator produces zero recall at high precision operating points, confirming its critical role beyond policy finetuning.

Datasets

• Windows Endpoint Detection Dataset — 490,184 samples (388,336 train, 59,162 validation, 42,686 test) — proprietary SOC data

Baselines vs proposed

• Direct-label Nemotron-3-Nano-30B SFT baseline: benign recall at high precision = 21.8% vs CoT+Calibration system: 64.8%
• Direct-label Nemotron-3-Nano-30B SFT baseline: malicious recall at high precision = 34.7% vs CoT+Calibration system: 53.0%
• Self-training only (iteration 25) policy: 81.7% accuracy vs full pipeline (self-training + RLVR): 82.6%
• RLVR without self-training: benign recall 61.6% vs full (+self-training) 64.8%
• Zero-shot general-purpose LLM (Claude Opus 4.6): accuracy 66.6% vs Nemotron-3-Nano-30B base: 65.8%, but trained model achieves 82.6%

Limitations

• Dataset limited to Windows endpoint detections; other platforms and sensor types not evaluated.
• Task framed as binary classification, no multi-class triage for threat categories or incident types.
• Proprietary and closed dataset limits external reproducibility and evaluation by outside researchers.
• Model and calibrator training details such as hyperparameters and seeds are partially unspecified.
• Distribution shift observed between validation and test benign precision, indicating sensitivity and need for ongoing recalibration.
• Confidence calibration approaches may not extend trivially to other LLM sizes or architectures without retraining.

Open questions / follow-ons

• How does the approach generalize to multi-class triage beyond the binary benign/malicious setting, e.g., mapping to specific threat classes or attack techniques?
• Can the method be extended and evaluated on other sensor modalities beyond Windows endpoints, such as network flows or cloud telemetry?
• How resilient is the calibrator and reasoning approach under adversarial distribution shifts or evasion attempts designed to confuse reasoning output?
• What are optimal methods for incremental or continual recalibration of confidence estimates during long-term deployment with changing attack landscapes?

Why it matters for bot defense

This paper’s methodology of coupling chain-of-thought reasoning with a dedicated confidence calibrator offers valuable lessons for bot-defense and CAPTCHA systems requiring interpretable, high-confidence classification decisions. Explicit reasoning enables auditability and analyst trust, key to reducing false positives and avoiding alert fatigue, just as in security operation centers. The finding that reasoning outputs degrade direct confidence estimates and thus require separate calibration is a critical insight, suggesting that CAPTCHA systems relying on raw LLM token probabilities may undervalue error. The multi-stage training approach combining prompt optimization, self-training, and reinforcement learning with verifiable rewards presents a replicable blueprint to boost model robustness and confidence quality. Bot-defense engineers can explore chain-of-thought and calibrator architectures to improve detection transparency and allow safe automated blocking decisions at high precision operating points. However, the domain specialization and data requirements underscore the importance of custom fine-tuning and careful calibration in reproducing these gains. Overall, the work encourages deeper integration of interpretable reasoning and rigorous confidence calibration in automated security classification use cases including bot detection and CAPTCHA solving.

Cite

@article{arxiv2607_28460,
  title={ Cybersecurity Detection Classification with Reasoning-enabled Language Models },
  author={ Amol Khanna and Manu Nandan and Cristian Viorel Popa and Joan Pujol-Roig and Diana Bolocan and Laura Vasilie and Alexandru Apostu and Chase Helwig and Mihaela Gaman and Michael Brautbar and Edward Raff and Chase Midler and Sven Krasser },
  journal={arXiv preprint arXiv:2607.28460},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28460}
}

Read the full paper

• arXiv:2607.28460 (abstract)
• arXiv:2607.28460 (PDF)