FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting

Source: arXiv:2607.14903 · Published 2026-07-16 · By Yanbing Shen, Fan Zhang, Haitao Xu

TL;DR

FirmPilot addresses the fragile and brittle nature of IoT firmware rehosting pipelines that aim to run firmware images in emulated environments for scalable dynamic analysis. Current template-driven approaches often fail to accommodate the diverse and complex vendor-specific boot scripts, persistent state, and network configurations required to bring firmware execution to an externally reachable and functional state. FirmPilot introduces an evidence-guided multi-agent environment recovery framework that iteratively reconstructs missing or broken environment artifacts through a closed-loop, execution-verified search and planning process. Key innovations include domain-specific agents tasked with filesystem/init artifact recovery, NVRAM persistent state synthesis, and network exposure configuration, all coordinated by a planner and informed by a search agent retrieving firmware-specific evidence from external sources. Evaluated on the large real-world LFwC corpus with 10,033 firmware images and the public FirmAE benchmark, FirmPilot significantly improves both network reachability (from 39.30% to 71.93%) and web-service reachability (from 25.49% to 52.39%) compared to the state-of-the-art FirmAE baseline. Moreover, the recovered rehosts support downstream security workflows such as RouterSploit exploitation and protocol-aware fuzzing, demonstrating that FirmPilot not only improves raw rehosting success but also the functional utility of rehosted firmware for security testing. The results highlight the effectiveness of evidence- and feedback-grounded multi-agent coordination in recovering cross-layer firmware environment dependencies that static template pipelines miss.

Key findings

• FirmPilot raises network reachability on the 10,033-image LFwC corpus from 39.30% to 71.93%, an absolute improvement of 3,274 images over the baseline FirmAE pipeline.
• Web-service reachability on LFwC improves from 25.49% to 52.39%, adding 2,699 more images with reachable management endpoints.
• On the 1,122-image public FirmAE benchmark, FirmPilot improves web reachability from 79.4% to 82.2%, showing consistent benefits even on templates that already cover many cases.
• Ablation experiments show that removing the PLAN agent costs 2,472 reachable images, NVRAM agent removal costs 2,003, Search agent removal costs 1,568, Network agent removal costs 1,558, and File agent removal costs 1,525 images.
• Persistent state synthesis recovers 479,440 NVRAM key entries across 1,971 images with rich domains including system, wireless, LAN, WAN, service exposure, and authentication states.
• Network exposure recovery materializes 5,370 exported IP entries, 2,703 QEMU network arguments, and 2,607 bridge/VLAN/interface setups across 5,191 images.
• A general-purpose coding agent baseline achieves only 5.43% web-service reachability on LFwC versus 52.39% with FirmPilot specialization.
• Downstream security workflows validate the practical utility of recovered services, including manually verified RouterSploit findings and protocol-aware fuzzing over detected service surfaces.

Threat model

The adversary is the complexity and heterogeneity of vendor-specific IoT firmware environments which prevent straightforward emulation and service reachability. The system must reconstruct missing boot, state, and network environment artifacts under a limited, partially observable setting without direct access to physical devices or full environment specifications. Large language models assist but must operate within bounded, auditable transitions to avoid non-reproducible changes. The rehosting process cannot guess unsupported or unrealistic state but relies on execution evidence and retrieved external knowledge constrained by runtime signals.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is the rehosting system facing diverse IoT firmware images with opaque vendor-specific boot and configuration conventions. The system assumes partial observability: only serial logs, device probes, and network reachability indicators are available at runtime; the actual device environment and state are unknown. The adversary here is the challenge of reconstructing a usable execution state in QEMU for each firmware under these uncertainties. External large language models (LLMs) assist but cannot directly rewrite environment artifacts without verification.

2. Data: The primary data is the LFwC corpus, a publicly released set of 10,033 locally executable IoT firmware images with vendor and architecture diversity. Metadata and ground-truth annotations are available for validation. The public FirmAE benchmark of 1,122 images is also used for comparison. Splits are fixed; all images are run under the same 2,400-second time budget per firmware execution. Labels include network reachability, web-service reachability, and downstream workflow success. Preprocessing involves unpacking, architecture detection, and baseline extraction typical of QEMU-based pipelines.

3. Architecture / Algorithm: FirmPilot decomposes the rehosting problem into an iterative multi-agent loop with five key agents: SEARCH (retrieves firmware-specific knowledge via similarity search from web and GitHub documentation), PLAN (schedules and sequences recovery actions based on observed evidence and dependencies), FILE (recovers boot/init filesystem artifacts through ranked candidates and typed startup deltas), NVRAM (synthesizes missing persistent key/value state overlays grounded in runtime reads and configuration evidence), and NETWORK (recovers network exposure via interface/IP binding and QEMU argument tuples).

Actions emitted by FILE, NVRAM, and NETWORK are typed deltas—structured and bounded environmental mutations rather than free-form edits. PLAN outputs an ordered sequence of such actions, which are applied and then validated via re-execution in the QEMU emulator. Feedback on execution outcomes (logs, probes, network services) closes the loop and refines the next retrieval/planning cycle. This process continues until no further admissible actions exist or success criteria are met.

4. Training Regime: The LLM backend (DeepSeek V4 Flash) is used for role-specific retrieval-augmented reasoning but no details on training epochs, batch size, or hyperparameters are given. The system itself executes firmware within fixed 2,400-second emulation budgets per image, allowing at most three planning rounds.

5. Evaluation Protocol: Metrics include ICMP ping reachability and web-service reachability defined as successful HTTP(S) transactions to exported guest-IPs including partial management page responses. Policies are consistent across baselines FirmAE and a general-purpose coding agent (Claude Code). Ablation experiments disable one agent at a time to quantify contribution. Statistical tests are not reported. Downstream workflows include service discovery (nmap), RouterSploit exploitation validation, and protocol-aware fuzzing on recovered services.

6. Reproducibility: LFwC is publicly released; FirmAE benchmark is public. Code release is not explicitly stated. The system relies on proprietary LLM backends. Artifact overlays and probe definitions are tightly integrated with QEMU. The extensive audit records attach provenance for all actions and evidence. End-to-end deterministic reproduction is not explicitly discussed but the typed action and execution-accepted delta model constrains non-determinism.

Concrete example: For an ASUS DSL-N10_C1 firmware, the full run reconstructs 198 NVRAM keys across LAN, WAN, UPnP, and interface domains, successfully starts the httpd daemon, and passes both ping and web probes; the no-NVRAM ablation run fails to reach the web service, showing the tangible effect of persistent state recovery.

Technical innovations

• Treating firmware rehosting as iterative evidence-indexed environment reconstruction, modeling artifact state transitions as bounded, typed deltas validated by repeated QEMU execution.
• A multi-agent architecture separating retrieval (SEARCH), planning (PLAN), and specialized artifact mutators (FILE, NVRAM, NETWORK) with strict role contracts and typed action interfaces to maintain reproducibility and bounded state changes.
• Use of retrieval-augmented generation combined with execution-grounded planning to resolve cross-layer dependencies across boot scripts, persistent configuration, and network setup preventing partial or fragile firmware executions.
• Large-scale ablation and evaluation on a 10,033-image real-world IoT firmware corpus demonstrating measurable contributions from each specialized agent to multi-dimensional recovery success.

Datasets

• LFwC — 10,033 images — public large-scale Linux firmware corpus with metadata and vulnerability research annotations
• FirmAE benchmark — 1,122 images — public firmware benchmark aligned with FirmAE templates and architectures

Baselines vs proposed

• FirmAE: web-service reachability = 25.49% on LFwC vs FirmPilot: 52.39%
• FirmAE: network reachability = 39.30% on LFwC vs FirmPilot: 71.93%
• FirmAE benchmark: web-service reachability = 79.4% vs FirmPilot: 82.2%
• General-purpose coding agent baseline (Claude Code): web-service reachability = 5.43% on LFwC vs FirmPilot: 52.39%
• No PLAN agent ablation: web-service reachability loss of 2,472 images on LFwC
• No NVRAM agent ablation: web-service reachability loss of 2,003 images on LFwC
• No SEARCH agent ablation: web-service reachability loss of 1,568 images on LFwC
• No NETWORK agent ablation: web-service reachability loss of 1,558 images on LFwC
• No FILE agent ablation: web-service reachability loss of 1,525 images on LFwC

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14903.

Fig 1: System Architecture of FIRMPILOT

Fig 2: Design of the SEARCH Agent

Fig 3: Design of the PLAN Agent

Fig 4: Design of the FILE Agent

Fig 5: Design of the NVRAM Agent

Fig 6: Design of the NETWORK Agent

Fig 7: LFwC Ping/Web Success by Firmware Architecture

Fig 8 (page 4).

Limitations

• Remaining failures stem from runtime emulation (2,604 images), extraction mismatches (1,675), and architecture handling (498), indicating upstream challenges outside recovery loop.
• LLM backend (DeepSeek V4 Flash) specifics including training parameters and robustness to noisy logs are not described; dependency on proprietary model could limit reproducibility or generalization.
• Evaluation lacks explicit adversarial robustness testing; e.g., no stress tests against deliberately malformed firmware or environment perturbations.
• Partial observability sets inherent limits on inference accuracy; some vendor-specific behaviors may remain irrecoverable without physical device feedback.
• No discussion of real-time or incremental rehosting—system is designed for offline bulk analysis with up to 2,400s emulation budgets.
• Code and artifacts release is not explicitly stated, creating potential barriers to independent reproduction.

Open questions / follow-ons

• How to extend environment recovery to support live dynamic updates or incremental rehosting for continuous firmware analysis?
• What adversarial or robustness guarantees can be formalized for evidence-guided recovery in hostile or corrupted firmware environments?
• Can the approach be generalized beyond router-class firmware to other IoT or embedded device types with significantly different init and networking conventions?
• How to reduce reliance on proprietary LLM backends or enable training specifically tailored for firmware rehosting semantic understanding?

Why it matters for bot defense

FirmPilot’s multi-agent evidence-guided environment reconstruction approach provides valuable insights to bot-defense and CAPTCHA practitioners regarding the challenge of reconstructing complex operational states under partial observability and heterogeneous inputs. The strict separation of semantic inference (via LLM-guided retrieval and planning) from deterministic, typed environment transitions, and validation through repeated black-box execution parallels the need in bot defense to ground decisions in verifiable signals and avoid heuristic drift. Just as FirmPilot uses firm causal traces and bounded mutation spaces to avoid irreproducible or unrealistic firmware state, CAPTCHA systems could adopt similar evidence-indexed loops to iteratively refine interaction models or adapt heuristics, ensuring reliable detection without overfitting or blind spots. The notion of coordinating specialized agents with role-specific, constrained action surfaces is also applicable to distributed bot detection that integrates multi-sensor inputs and feedback loops. Overall, FirmPilot exemplifies a hybrid human-comprehensible plus execution-verified approach to automating environment recovery that may inspire more principled, auditable bot detection methods combining ML inference with trusted domain-specific constraints.

Cite

@article{arxiv2607_14903,
  title={ FirmPilot: Evidence-Guided Multi-Agent Environment Recovery for IoT Firmware Rehosting },
  author={ Yanbing Shen and Fan Zhang and Haitao Xu },
  journal={arXiv preprint arXiv:2607.14903},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14903}
}

Read the full paper

• arXiv:2607.14903 (abstract)
• arXiv:2607.14903 (PDF)