HTTP REST API Structure Learning
Source: arXiv:2607.02442 · Published 2026-07-02 · By Ran Dubin, Amit Dvir
TL;DR
This paper addresses the challenge of securing HTTP REST APIs by learning their structural behavior directly from observed network traffic, without relying on potentially incomplete or unavailable OpenAPI documentation. The authors propose HRAL (HTTP REST API Learning), an unsupervised anomaly detection method that models API endpoints by clustering request paths, extracting path parameters, and enriching these models into OpenAPI-compliant specifications. This learned structure serves as a behavioral baseline to detect anomalous API requests indicative of attacks. HRAL is evaluated on the ATRDF dataset, containing 18 API endpoints and seven attack types, and is compared to existing baselines including Speculator and supervised detection methods with varying OpenAPI documentation completeness. The results show HRAL achieves a strong average recall of 82.07% and F1-score of 87.24%, significantly outperforming unsupervised Speculator and minimally or partially documented OpenAPI baselines; its performance approaches that of full OpenAPI documentation. Furthermore, combining HRAL with signature-based rules from OWASP ModSecurity CRS achieves 100% detection coverage in controlled experiments. Overall, HRAL proves valuable in real-world scenarios where documentation may be missing or outdated, offering a robust foundation for detecting structural API anomalies and augmenting existing security tools.
Key findings
- HRAL achieves an average recall of 82.07% and F1-score of 87.24% across 7 attack types on ATRDF dataset without requiring API documentation.
- Full OpenAPI documentation baseline achieves 93.57% average recall, outperforming HRAL by about 11.5%, but HRAL outperforms Minimal (22.94%) and Basic (72.30%) documentation baselines.
- Speculator unsupervised baseline achieves only 23.84% average recall and completely fails to detect attacks like Cookie Injection, Log Forging, SQL Injection, and XSS.
- HRAL detects all attack types robustly, including those poorly detected by Speculator, with recall per attack type often above 90%, except Log4j (37.17%).
- Combining HRAL with signature-based detection rules (e.g., OWASP ModSecurity CRS) leads to 100% detection recall across all attack vectors on ATRDF.
- HRAL's clustering uses Ward linkage with a distance threshold of 2, chosen empirically via silhouette score optimization.
- HRAL enriches path parameter modeling by collecting statistics on parameter length, type, and regex patterns, improving endpoint specification accuracy.
- HRAL can reconstruct or enhance OpenAPI specifications solely from HTTP request traffic, enabling API understanding even with missing or partial documentation.
Threat model
The adversary is a malicious actor who crafts API requests to exploit vulnerabilities such as injection attacks or unauthorized access. They can generate requests mimicking normal API behavior but aim to introduce anomalous structural patterns to compromise the system. The adversary cannot completely mask structural deviations inherent in malicious requests if normal API behavior is well modeled. They do not have full knowledge of the learned model or the ability to inject perfectly legitimate-looking traffic at all times.
Methodology — deep read
Threat Model & Assumptions: The adversary is assumed to perform malicious API requests, including injection attacks and exploitation attempts, aiming to bypass security by mimicking normal API structures. The defender aims to model normal API request structures to detect anomalies. The adversary cannot fully evade detection if the structure deviates significantly.
Data: The authors use the ATRDF dataset, a publicly available API traffic dataset comprising 18 diverse API endpoints and labeled benign and attack traffic with 7 attack types (Cookie Injection, Directory Traversal, Log Forging, Log4j, Remote Code Execution, SQL Injection, XSS). The dataset contains roughly 108k normal and 109k abnormal samples.
Architecture / Algorithm: HRAL processes HTTP requests using a sequence of steps. First, textual elements of requests (paths, headers, query params) are stemmed with Porter Stemmer and vectorized using CountVectorizer. Then, Agglomerative Hierarchical Clustering with Ward linkage and distance threshold 2 groups requests into clusters corresponding to API endpoints. Within clusters, the algorithm identifies static parts and variable path parameters by comparing paths. Parameters are statistically characterized (lengths, types, regex patterns). These enriched clusters are passed to the Speculator tool, which automatically constructs an OpenAPI-compliant specification from structured endpoint data, improved with HRAL's parameterization step that overcomes Speculator's limitations in parameter detection.
Training Regime: HRAL is unsupervised, requiring no labeled training data. Parameter choices such as clustering threshold (2) are selected empirically via silhouette score on a held-out subset of training traffic. No traditional training epochs or optimization is involved since this is a clustering and rule-extraction pipeline.
Evaluation Protocol: HRAL's anomaly detection performance is evaluated by computing recall and F1-score on detecting malicious API requests in ATRDF. Baselines include Minimal, Basic, and Full OpenAPI documentation-based anomaly detectors (supervised), and the Speculator unsupervised specification builder. Evaluation considers detection per attack type and overall, reporting recall and F1 scores in detail (see Table III). Additional tests combine HRAL with OWASP ModSecurity CRS signature rules to assess combined detection coverage. The evaluation is conducted on unseen API traffic to simulate real deployment scenarios.
Reproducibility: ATRDF is publicly available, enabling replication. Speculator is open source. HRAL is described algorithmically with pseudocode; however, code availability is not explicitly stated. Exact random seeds and hardware details are not provided, suggesting partial reproducibility.
One concrete example: Consider requests to "/bookstore/{username}/account info" endpoint appearing with varying usernames. HRAL groups these paths into one cluster, detects "username" as a path parameter by comparing paths across requests, and generates regex patterns and length constraints from observed data. This parameterization refines the OpenAPI spec, enabling detection of anomalous deviations from expected username patterns in future requests.
Technical innovations
- Unsupervised clustering-based algorithm that groups HTTP REST API requests by endpoint using hierarchical agglomerative clustering with adaptively chosen threshold.
- Novel parameterization method to detect and characterize string-based path parameters using statistical analysis, improving upon Speculator's naive parameter treatment.
- Capability to generate enriched OpenAPI-compliant specifications solely from request traffic without requiring any prior documentation.
- Hybrid security approach that combines structural anomaly detection (HRAL) with signature-based rules (e.g., OWASP ModSecurity CRS) to achieve full attack detection coverage.
Datasets
- ATRDF — approximately 217,000 HTTP requests — public API security dataset with 18 endpoints, 7 attack types, benign and malicious labels.
Baselines vs proposed
- Minimal OpenAPI documentation: average recall 22.94% vs HRAL: 82.07%
- Basic OpenAPI documentation: average recall 72.30% vs HRAL: 82.07%
- Full OpenAPI documentation: average recall 93.57% vs HRAL: 82.07%
- Speculator unsupervised baseline: average recall 23.84% vs HRAL: 82.07%
- HRAL combined with OWASP ModSecurity CRS rules: detection recall improved to 100%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02442.
Fig 1: OpenAPI Request-Based Algorithm
Fig 2: OpenAPI Request-based and Spec-building Algorithm
Limitations
- Scalability and performance: clustering a high volume of high-dimensional HTTP requests is computationally intensive and may not scale to real-time or large-scale deployments.
- Adaptability: structural API changes (new endpoints, schema modifications) require retraining or reclustering, adding operational overhead and latency.
- Interpretability: as an unsupervised clustering method, HRAL lacks explainability of anomaly causes, limiting analyst insight and remediation.
- Detection blind spots: attacks embedded in request bodies rather than URL paths or query parameters (e.g., Log4j payload injection) are less effectively detected.
- Partial reproducibility: code and exact training/evaluation environment details are not fully disclosed.
- Evaluation limited to ATRDF dataset; cross-dataset generalization not tested due to lack of similar public datasets.
Open questions / follow-ons
- How can HRAL be extended or combined with payload (request body) analysis to detect payload-based attacks like Log4j more effectively?
- What strategies can improve HRAL’s scalability and runtime efficiency to enable large-scale, low-latency deployment?
- How to incorporate interpretability and explainability into HRAL to provide actionable insights for security analysts?
- Could hybrid models combining unsupervised structural learning with supervised fine-tuning improve detection and adaptability to evolving APIs?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, HRAL’s approach to unsupervised learning of API endpoint behavior provides a strong method to detect anomalous API usage that could indicate automated or malicious client behavior. Unlike approaches relying on static or partial API documentation, HRAL can adapt to undocumented or partially documented APIs—common in large-scale, dynamic web services. This capability complements CAPTCHA-based protections by identifying suspicious API requests structurally atypical from normal usage, enabling earlier or more precise mitigation. Furthermore, combining structural anomaly detection with signature-based rules, as HRAL demonstrates, can close detection gaps important for defending APIs exposed to bot and injection attacks. Bot-defense engineers should consider integrating API structure learning mechanisms like HRAL into layered defenses, especially when documentation is unreliable or absent. However, attention must be paid to scalability challenges and detection limits on payload-based threats beyond URL and query patterns.
Cite
@article{arxiv2607_02442,
title={ HTTP REST API Structure Learning },
author={ Ran Dubin and Amit Dvir },
journal={arXiv preprint arXiv:2607.02442},
year={ 2026 },
url={https://arxiv.org/abs/2607.02442}
}