Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

Source: arXiv:2607.16175 · Published 2026-07-17 · By Md Erfan, Ahmed Ryan, Md Kamal Hossain Chowdhury, Md Rayhanur Rahman

TL;DR

This paper addresses the challenge of converting unstructured vulnerability descriptions for Connected and Autonomous Vehicles (CAVs) into structured threat information using the STIX (Structured Threat Information Expression) format. Unlike conventional CVE text entries that are human-readable but unstructured, security practitioners need machine-readable structured data capturing affected assets, weakness types (CWEs), and attack techniques (MITRE ATT&CK) to prioritize defenses effectively. The authors create a novel, manually curated dataset called CAV-STIXGen by collecting 183 CAV-related CVEs from the National Vulnerability Database, annotating them with STIX domain objects (SDO), STIX relationship objects (SRO), CWE classifications, and MITRE ATT&CK techniques. They then evaluate 11 open-weight Large Language Models (LLMs) ranging from 4B to 120B parameters on the task of generating STIX JSON bundles from CVE texts using multiple prompting strategies. The best single-model results achieve F1 scores of up to 0.94 for SDO extraction and 0.63 for SRO extraction, and nearly perfect CWE mapping at 0.99 F1, though complete MITRE ATT&CK mapping remains challenging. Multi-agent approaches combining different models show complementary gains. Additionally, the paper analyzes the frequent CWE and ATT&CK patterns in CAV vulnerabilities, revealing common threat behaviors and co-occurring attack techniques. Overall, this work provides a new benchmark resource and detailed empirical evaluation demonstrating that open-weight LLMs can generate meaningful structured cyber threat intelligence representations from vehicle-related vulnerability descriptions, highlighting progress and current limitations of AI-assisted automation in automotive security context.

Key findings

• The CAV-STIXGen dataset contains 183 manually validated CAV-relevant CVEs, with 1,383 STIX domain objects and 1,395 STIX relationship objects annotated.
• Under dynamic few-shot prompting, Phi-4 achieves an SDO extraction F1 score of 0.94, while Qwen-Coder achieves the highest SRO extraction F1 of 0.63.
• CWE mapping is highly accurate with multiple models reaching F1 scores near 0.99 using few-shot prompting.
• MITRE ATT&CK mapping remains difficult: top Match@1 scores reach 0.68 (Gemma-4-31B) but Match@All scores are substantially lower, showing incomplete technique identification.
• Multi-agent configurations combining Gemma-4-31B and Codestral-22B achieve SDO F1 of 0.91 and SRO F1 of 0.43, showing some gains from role decomposition.
• The dataset covers 10 of 19 STIX domain object types with infrastructure (538 instances) and attack-pattern objects being most frequent.
• Frequent CWEs include CWE-787 (out-of-bounds write) and CWE-20 (improper input validation), aligning with CWE Top 25 known weaknesses.
• Most frequent MITRE ATT&CK technique is T1499 (endpoint denial of service), with frequent co-occurrence of T1203 (exploitation of client execution) and T1499 pairs appearing in 10 CVEs.

Threat model

The adversary is not explicitly modeled as a malicious actor against the LLMs but rather the threat context is a security analyst needing accurate structured threat intelligence for CAV vulnerabilities. The study assumes no adversarial manipulation of CVE inputs or model behaviors. Adversaries exploit vulnerabilities documented in CVEs; the focus is on automating intelligence extraction, not on resisting direct attacks on the extraction method.

Methodology — deep read

The study follows a structured multi-step methodology:

1. Threat Model and Assumptions: The focus is on automated extraction of structured threat info from unstructured CVE descriptions related to CAVs. The threat is realistic use by security practitioners to analyze vulnerabilities. The adversary here is implicit (attacker exploiting vulnerabilities). The paper does not explicitly adversarially test the models against evasion or poisoning, it evaluates LLM generation quality.

2. Data Collection and Annotation: The authors constructed the CAV-STIXGen dataset by querying the National Vulnerability Database (NVD) via REST API using domain-specific keywords filtered and refined manually. From 445 candidate CVEs (2012-2025), after relevance filtering by annotators (94% inter-rater agreement), 183 CVEs relevant to CAV components were retained.

Each CVE was manually annotated for STIX Domain Objects (SDO) across 10 object types (e.g., infrastructure, attack-pattern, vulnerability, identity), STIX Relationship Objects (SRO), Common Weakness Enumeration (CWE) mappings (manually verified), and MITRE ATT&CK technique mappings. Relationships include types such as exploits, targets, affects, uses.

Annotations were recorded in CSV files for review, consensus, and subsequently programmatically converted to STIX 2.1 JSON bundles. The final dataset includes 1,383 SDOs, 1,395 SROs, 211 CWE mappings, and 294 MITRE ATT&CK technique mappings.

3. Model Evaluation Setup: Eleven open-weight LLMs ranging from 4B to 120B parameters were evaluated, categorized as general-purpose, cybersecurity-specialized, code-oriented, and large-capacity open-weight models. Models include Gemma, Phi, Qwen, Codestral, LLaMA, GPT-OSS, among others.

Each model received a CVE identifier and description, and generated a STIX JSON bundle output. Evaluation compared generated objects, relationships, CWE, and ATT&CK mappings to ground-truth with micro-precision, recall, and F1 metrics.

Prompting strategies were tested: (a) Contextless prompting with CVE as input; (b) STIX-guided prompting adding explicit task instructions on object/relationship types and mappings; (c) Dynamic few-shot prompting adding five relevant ground-truth STIX examples selected by similarity.

Experiments considered 4 temperature settings (0, 0.25, 0.75, 1.0) to vary randomness.

4. Evaluation Metrics and Protocol: Object extraction, relationship extraction, CWE mapping scored with micro-averaged precision, recall, F1. MITRE ATT&CK was evaluated with Match@1 (at least one correct) and Match@All (all correct) metrics. Results were sliced by prompting strategy, model, and subtasks for fine-grained analysis.

5. CWE and MITRE ATT&CK Analytical Methods: Frequency analysis identified common weakness types and attack techniques. Co-occurrence mining of ATT&CK techniques used FP-Growth frequent itemset mining with support threshold 0.02 and association rules with confidence 0.3, to find recurring attack combinations.

6. Experimental Environment: Evaluation ran on a local Ubuntu server with LM Studio API hosting models on four NVIDIA RTX 6000 Ada GPUs totaling 192GB VRAM.

Concrete Example: For CVE-2023-289011, describing a broken access control vulnerability in Skoda Automotive Cloud, annotators extracted multiple SDOs (vendor, cloud, service, vulnerability, attacker), linked them with SROs (exploits, targets), assigned CWE-200 and ATT&CK technique T1190, and created the STIX JSON bundle. Models then attempted to recreate this structured STIX output from the CVE text under various prompting conditions.

Reproducibility: The dataset, prompts, and scripts are publicly released on Figshare. All models evaluated are open-weight and run locally. Exact seeds/hyperparameters for generation are partially described but no frozen weights or closed datasets were involved.

Technical innovations

• Construction of CAV-STIXGen, the first manually annotated dataset linking CAV domain CVE descriptions with STIX 2.1 domain and relationship objects, CWE categories, and MITRE ATT&CK techniques.
• Comprehensive evaluation of 11 open-weight LLMs (4B-120B) on multi-task structured CTI generation from unstructured vulnerability text using detailed micro-level precision/recall/F1 metrics per subtask.
• Introduction of dynamic few-shot prompting guided by similarity scores to inject relevant STIX examples, substantially improving model performance on complex relationship extraction and semantic mappings.
• Multi-agent model configuration combining general and code-specialized LLMs to explore role-based decomposition for improved STIX object and relationship extraction.

Datasets

• CAV-STIXGen — 183 annotated CVEs with 1,383 STIX domain objects and 1,395 STIX relationships — constructed from NVD CVE database (2012-2025)

Baselines vs proposed

• Gemma-4-31B (dynamic few-shot): SDO F1 = 0.86 vs Codestral-22B (dynamic few-shot): SRO F1 = 0.46
• Phi-4 (dynamic few-shot): SDO F1 = 0.94 vs Qwen-Coder (dynamic few-shot): SRO F1 = 0.63
• Multiple models (dynamic few-shot): CWE mapping F1 ≈ 0.99, showing near-perfect classification
• Gemma-4-31B (dynamic few-shot): ATT&CK Match@1 = 0.68 vs Qwen-Coder: 0.67 (best scores), Match@All substantially lower (~0.2–0.3)
• Multi-agent (Gemma + Codestral) at 0.75 temperature: SDO F1 = 0.91 vs SRO F1 = 0.43, showing some improvement over single-model SRO extraction

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16175.

Fig 1: An Example of CVE Description to STIX Generation.

Fig 2: Prompting strategies for CVE-to-STIX generation.

Limitations

• The dataset covers only 183 CVEs focused on CAV-related software and hardware components, limiting generalizability to other domains.
• Evaluation does not test adversarial robustness; model outputs could be vulnerable to hallucinations or crafted inputs.
• MITRE ATT&CK full mapping remains weak, with low Match@All indicating incomplete and partial attack technique extraction.
• Multi-agent approach gains are modest and limited to two models; scaling this paradigm is unexplored.
• Relationship extraction remains challenging with lower F1 (~0.6 max) compared to domain object extraction (~0.94 F1).
• Dynamic few-shot prompting relies on quality and availability of similar examples; scaling to zero-shot or unfamiliar CVEs may degrade performance.

Open questions / follow-ons

• How do the evaluated LLMs perform on zero-shot CVE-to-STIX tasks with previously unseen or out-of-domain vulnerabilities?
• Can multi-agent or ensemble approaches combining diverse open-weight LLMs improve full MITRE ATT&CK technique coverage significantly beyond the current low Match@All scores?
• What techniques can mitigate hallucination and enhance relationship extraction quality, especially for semantically rich STIX relationships?
• How transferable is this automated STIX generation methodology to other safety-critical domains beyond CAVs, such as industrial control systems or medical IoT?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focusing on cybersecurity automation, this work highlights how open-weight LLMs can be leveraged to extract structured threat intelligence from unstructured textual vulnerability databases specific to highly complex domains like autonomous vehicles. The detailed evaluation of prompting methods and model architectures provides insights into balancing generative AI capabilities with domain-specific knowledge. Additionally, the paper shows that while LLMs are strong at recognizing explicit objects and mapping weaknesses (CWEs), relational reasoning and comprehensive mapping of attacker techniques remain challenging, pointing to potential limitations when integrating such AI systems into automated defense pipelines reliant on precise CTI structuring. Practitioners can consider incorporating few-shot example-guided prompting and multi-model ensembles to improve structured output quality. The constructed CAV-STIXGen dataset is a valuable resource for benchmarking structured extraction tasks in automotive cybersecurity contexts and may inspire similar dataset efforts in bot defense threat intelligence. Finally, the frequent CWE and ATT&CK co-occurrence analyses exemplify how automated structured threat data can reveal attack patterns to prioritize protections, which is relevant for CAPTCHA systems protecting vehicle telematics or infrastructure access points.

Cite

@article{arxiv2607_16175,
  title={ Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities },
  author={ Md Erfan and Ahmed Ryan and Md Kamal Hossain Chowdhury and Md Rayhanur Rahman },
  journal={arXiv preprint arXiv:2607.16175},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16175}
}

Read the full paper

• arXiv:2607.16175 (abstract)
• arXiv:2607.16175 (PDF)