Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer
Source: arXiv:2606.21071 · Published 2026-06-19 · By Zhengsong Zhang, Zongze Li, Jiawei Guo, Haipeng Cai
TL;DR
This paper addresses the under-explored security risks in the implementation layer of local large language model (LLM) agents—software runtimes like OpenClaw and Nanobot that execute model-driven actions on user hosts with broad privileges. Unlike previous work that focused on prompt injection, malicious skills, or black-box behavior, this study audits the agent runtime’s source code to find vulnerable design and implementation patterns that enable privilege escalation, injection, or leakage. The authors derive a principled vulnerability taxonomy guided by STRIDE, covering prompt handling, tool/skill execution, permission/isolation, network communication, and authentication/authorization. They encode 77 domain-specific static analysis rules (47 Semgrep, 30 CodeQL) targeting these patterns and evaluate them on a benchmark of 446 source-level advisories from OpenClaw, temporally split for training and testing. The tailored rules significantly increase recall of known vulnerabilities compared to generic static-analysis baselines, showing that the agent runtime is a rich source of implementation bugs exploitable by adversaries. However, the rules produce many findings requiring manual triage, motivating future semantic filtering. The work pioneers systematic source-level auditing to complement behavioral and prompt-level defenses for local LLM agents running with user privileges.
Key findings
- CLAWAUDIT’s Semgrep rules raise recall on held-out 217 test advisories from 21.7% (Pro baseline) to 66.8%, a +45.1 percentage points improvement.
- CLAWAUDIT’s CodeQL queries raise recall on the test partition from 13.8% (security-extended baseline) to 75.1%, a +61.3 percentage points gain.
- Train/test recall gaps remain under 4 percentage points, indicating rules generalize well to unseen vulnerabilities.
- OPENCLAWBENCH contains 446 advisories with 12 critical, 134 high, 250 medium, and 50 low severity labels, spanning 84 CWE classes.
- Permission & Isolation (CAT-3) and Network & Communication (CAT-4) categories contain the largest numbers of domain-specific audit rules (14 Semgrep + 12 CodeQL for CAT-3, 13 Semgrep + 7 CodeQL for CAT-4).
- Generic static analyzers detect less than one-third (31.3%) of disclosed vulnerabilities when combined, highlighting coverage gaps for local LLM agent runtimes.
- Live-code scanning on current OpenClaw versions with CLAWAUDIT rules generates many findings requiring manual triage; thus, high recall comes with low precision before semantic filtering.
Threat model
The adversary is an external user or attacker controlling untrusted natural language input to a local LLM agent runtime running with user privileges on the host machine. They can manipulate prompt inputs, model outputs, or loaded skills to attempt unauthorized privilege escalations, command injections, data exfiltration, or bypasses of permission gates. The adversary cannot arbitrarily modify the agent’s source code but can exploit vulnerabilities in the runtime's mediation components. Denial-of-Service and repudiation attacks are excluded from scope due to static analysis limitations.
Methodology — deep read
The authors begin by identifying the agent runtime layer as the privileged software boundary mediating untrusted natural language inputs and high-privilege host actions, including prompt building, parsing model output, dispatching tools/skills, memory operations, network calls, and permission checking (Section 2.1).
They establish a threat model based on STRIDE, excluding denial-of-service and repudiation as out of scope for static source analysis. The four covered categories are Tampering (split into 1) prompt handling and 2) tool/skill execution), Elevation of Privilege (permission & isolation), Information Disclosure (network & communication), and Spoofing (authentication & authorization) (Section 3).
To build practical audit rules, they construct a data-flow diagram of the OpenClaw runtime and identify trust boundaries where untrusted data crosses into privilege-sensitive operations. For each, they answer what untrusted data crosses, what privileged operations it triggers, and which source-level component mediates this transition, forming the basis for a mutually exclusive five-category taxonomy.
Using 229 vulnerabilities from OpenClaw GitHub Security Advisories dated before 2026-04-01, they manually inspect vulnerable code and extract recurring implementation bug patterns. These patterns inform handcrafted static-analysis rules implemented in two backends: Semgrep (AST template matching with metavariables and negations) and CodeQL (declarative predicates over relational program graphs with data-flow reasoning) (Section 4).
The semantic equivalence of rules across both platforms enables measuring improvements due to taxonomy/rule design versus backend capability. This results in 47 Semgrep YAML rules and 30 CodeQL queries covering the five categories.
They collect a benchmark OPENCLAWBENCH with 446 advisories total, split by date into 229 train advisories and 217 held-out tests. Each advisory includes vulnerable source snapshots, fix commits, modified files, severity, CWE labels, and assigned taxonomy categories (Section 5).
Evaluation measures advisory-level recall: an advisory is detected if any rule produces a finding within any file modified by its fix commit. They run Semgrep Pro baseline rules (≈2,815 community rules) and CodeQL javascript-security-extended suite with and without CLAWAUDIT rules on all vulnerable snapshots, totaling 446 runs each. Scan budgets are 2 minutes per Semgrep run and up to 1 hour per snapshot for CodeQL. They also report train/test recall separately to measure generalization.
A live-code audit scanning the full OpenClaw repository (over 16K files) evaluates findings per query to assess triage burden and precision limitations.
This approach walks through one concrete example end-to-end: an advisory involving a path traversal vulnerability patched in certain JavaScript files is detected by CLAWAUDIT’s permission & isolation category rules matching code patterns that enforce sandbox boundaries.
Limitations include the static nature of the analysis (does not detect runtime-only or semantic flaws), reliance on file-level labeling rather than line-level matches for recall, and the focus on OpenClaw which may limit generality. The large number of findings in live scans show rules alone do not guarantee deployable precision.
Technical innovations
- Deriving an agent-runtime-specific vulnerability taxonomy from STRIDE threat categories tailored to local LLM agents’ unique mediation components.
- Encoding 77 domain-specific static analysis rules (47 Semgrep YAML, 30 CodeQL queries) targeting privileged runtime components such as prompt builders, tool dispatchers, and permission gates.
- Building OPENCLAWBENCH, a large, temporally split benchmark of 446 source-code-level advisories from the OpenClaw repository for training and evaluation of agent runtime audit rules.
- Demonstrating that domain-specific rules substantially improve recall of known vulnerabilities over generic static analysis rule sets on the same code corpus.
- Comparing two static analysis backends with syntactic and relational paradigms using parallel rule sets to separate taxonomy effectiveness from analysis engine capabilities.
Datasets
- OPENCLAWBENCH — 446 source-code advisories — collected from OpenClaw GitHub Security Advisories (public)
- OpenClaw source code repository — >16,000 JavaScript/TypeScript files — public on GitHub
Baselines vs proposed
- Semgrep Pro baseline: recall = 21.7% on test set vs. Semgrep + CLAWAUDIT recall = 66.8%
- CodeQL security-extended baseline: recall = 13.8% on test set vs. CodeQL + CLAWAUDIT recall = 75.1%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.21071.
Fig 1: Overview of CLAWAUDIT’s taxonomy→rule→benchmark pipeline. The framework derives an
Limitations
- Static analysis cannot detect vulnerabilities depending on runtime workload, model semantic judgment, or dynamic environment.
- File-level attribution of findings may both overcount (unrelated patterns in fixed files) and undercount (vulnerabilities spread across files) detections.
- Benchmark only covers OpenClaw repository vulnerabilities; generalization to other local LLM agents is untested.
- Low precision and high volume of false positives on live code repositories require substantial manual triage prior to production use.
- Denial-of-Service and Repudiation threats fall outside the scope due to static analysis limitations.
- Rule writing depends on publicly disclosed advisories, possibly missing novel bug classes or attacker techniques.
Open questions / follow-ons
- How can semantic filtering or dynamic analysis be integrated with static rules to reduce false positive triage cost in live deployments?
- What new vulnerabilities emerge in other local LLM agents with different architectures not covered by OpenClaw-based rules?
- How do runtime-only faults or model-driven semantic misuse impact agent safety beyond the static vulnerabilities detected here?
- Can automated or learning-based methods assist in generating more precise static rules or prioritizing findings in the agent runtime context?
Why it matters for bot defense
Bot-defense and CAPTCHA practitioners developing or deploying local LLM agents should treat the agent runtime as a critical security boundary beyond simply aligning models or vetting skills. This work underscores that vulnerabilities in the prompt builder, tool dispatcher, permission gates, and network client can directly enable privilege escalation, command injection, SSRF, or data leaks on the host. Practical defenses thus require rigorous static auditing of agent runtimes using domain-specific patterns like those presented here to complement behavioral or prompt-level filters. However, the reported volume of findings signals the need for semantic triage layers to maintain manageable operational workflows. Overall, integrating such source-aware static audits into the SDLC can harden local agent deployments against adversaries aiming to abuse runtime mediation components.
Cite
@article{arxiv2606_21071,
title={ Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer },
author={ Zhengsong Zhang and Zongze Li and Jiawei Guo and Haipeng Cai },
journal={arXiv preprint arXiv:2606.21071},
year={ 2026 },
url={https://arxiv.org/abs/2606.21071}
}