The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting
Source: arXiv:2607.23710 · Published 2026-07-26 · By Ishpuneet Singh, Shreyas Mahajan, Gurjot Singh, Maninder Singh
TL;DR
This paper rigorously evaluates the security of user authentication code generated by five prominent AI coding assistants (GitHub Copilot with Claude Sonnet 4.5 and 4.6, OpenAI Codex GPT 5.3, Google Antigravity Gemini 3.1 Pro, and Cursor IDE free tier). The authors focus exclusively on authentication workflows implemented as Flask applications with raw SQL (to stress test security without ORM guardrails). They assess four prompting strategies representing different levels of developer expertise and guidance: Basic (novice), Secure (generic security prompt), NIST-Based (explicit embedding of NIST SP 800-63B guidelines), and iterative Reprompting (auto self-audit and refinement using NIST context). Combining manual static code review with dynamic penetration testing, they map the generated code against a unified evaluation matrix synthesizing NIST and OWASP authentication controls. The empirical results show that basic or generic security prompts yield code lacking critical protections (brute force resistance, session management, password handling). Explicit single-shot NIST prompting improves compliance but is still structurally insufficient. Only iterative reprompting enables comprehensive defense-in-depth security architectures consistently across all models. The study concludes that current LLM code generation tools produce insecure-by-default authentication code, mandating a shift in enterprise workflows towards continuous, standards-driven verification involving iterative prompting and auditing.
Key findings
- In Basic Prompt (BP) mode, on average models implemented only about 11-15 of 34 security parameters across severity levels, frequently omitting brute-force rate limiting, account lockout, and strong password policies.
- Secure Prompt (SP) gave modest improvements over BP but still missed critical protections like CAPTCHA on failed logins or explicit work factor configuration for hashing.
- NIST-Based Prompt (NP) improved compliance further, with some models reaching up to 25.5/34 parameters implemented (Fig 2), adopting strong KDFs (PBKDF2/Argon2) and enforcing password blocklists.
- Iterative Reprompting (RP) led to best compliance scores, up to 30.5/34 parameters, consistently applying rate limiting, MFA support, session cookie flags (HttpOnly, Secure, SameSite), CSRF tokens, and inactivity timeouts.
- Dynamic testing confirmed practical exploitability of missing protections: lack of account lockout enabled brute force attacks; missing CSRF tokens allowed cross-site request forgery; lack of session management flags enabled session hijacking.
- Injection vulnerabilities were generally mitigated via parameterized queries even under raw SQL constraints, showing models internalized well the prevention of SQLi.
- Among models, performance varied slightly but no model was fully secure without iterative reprompting, and newer versions (e.g., Copilot Sonnet 4.6) showed incremental improvements.
- MFA support was only implemented consistently in Reprompting prompts, indicating that single-shot prompts are insufficient for multi-factor flows.
Threat model
The adversary is a remote attacker capable of conducting brute-force password guessing, session hijacking via cookie theft, cross-site request forgery attacks without user consent, and injection attacks such as SQL injection or XSS via hostile payloads. The attacker cannot compromise developer-side secrets, hardware tokens, or insider knowledge of internal server implementations. The focus is on attacking incorrectly implemented authentication mechanisms that fail standard protections.
Methodology — deep read
The study investigates whether modern AI coding assistants can autonomously generate secure authentication code adhering to contemporary industry standards (NIST SP 800-63B and OWASP). The threat model implicitly assumes a remote adversary capable of standard attack vectors like brute forcing, session hijacking, CSRF, and injection but not insider or implementation-specific insider knowledge.
Data and setup involved code generation constrained to Flask framework with SQLite3 using raw SQL queries — deliberately avoiding ORMs to remove framework-level protections and test model autonomous security knowledge. The experiments spanned five prominent AI coding tools: Google Antigravity (Gemini 3.1 Pro), Cursor IDE (free tier), OpenAI Codex (GPT 5.3), and GitHub Copilot with Claude Sonnet 4.5 and 4.6.
To simulate developers of various expertise, four prompt strategies were designed:
- Basic Prompt (BP): functional, no security guidance, simulates novice developer.
- Secure Prompt (SP): adds vague "secure and clean" wording, simulating intermediate guidance.
- NIST-Based Prompt (NP): appends extracted key NIST SP 800-63B authentication guidelines (via Google NotebookLM document extraction).
- Iterative Reprompting (RP): after code generation under NP prompt, models receive their own generated code plus the NIST context, asked to self-audit listing fulfilled and missing security controls and iteratively refine code until fully compliant.
Each prompt completed an isolated new session without leakage to avoid contextual bias. Code generation produced a Flask app with four views (landing, registration, login, dashboard) and all necessary HTML templates. The exact prompts were documented to ensure reproducibility.
Evaluation combined manual static code review against a unified evaluation matrix synthesizing NIST SP 800-63B and OWASP rules covering password policy, MFA, session management, CSRF, HTTP headers, injection defense, and more. Static review identified presence, absence, or partial implementation of controls.
Dynamic penetration testing was performed on locally deployed apps running on Ubuntu 22.04 using Burp Suite and HackingBuddyGPT. Tests included brute-force login attempts (checking rate limits, account lockouts, response uniformity), session hijacking (cookie replay), CSRF exploitation (malicious HTML forms), and header inspection. Exploitability confirmed vulnerability beyond static absence.
Table 2 reports static and dynamic results per model-prompt combinations across ~34 parameters. Performance was scored as full, partial, or missing with numerical weights. Aggregate statistics and severity-stratified compliance are shown in Figures 2 and 3.
The study’s replicable discipline with detailed prompt disclosure, local deployment, manual and dynamic testing integration, and use of multiple competitive commercial/public AI models contributes a rigorous and transparent methodology. The step-by-step reprompting workflow exemplifies an iterative code generation and verification loop.
One concrete example: Under NP prompt, a GitHub Copilot (Sonnet 4.6) generated code that implemented strong password policies (min 15 chars, blocklist), used PBKDF2 hashing, set HttpOnly and Secure cookie flags, and applied inactivity timeouts. Yet, CSRF protections were incomplete. After RP (iterative reprompt), the model added missing CSRF tokens, enforced stricter session timeouts, and added account lockout with rate limiting, achieving a near-complete compliance score of 30.5/34.
Technical innovations
- Bi-modal security assessment combining manual static code review and dynamic penetration testing aligned with NIST SP 800-63B and OWASP, enabling practical exploit validation of AI-generated authentication code.
- Multi-tiered prompting strategy with iterative Reprompting forcing the LLM into a structured self-auditing and refinement loop using extracted authentication standards for comprehensive code hardening.
- Use of NotebookLM for targeted extraction of authentication-relevant directives from NIST SP 800-63B PDF to inform prompting, bridging document understanding and code generation.
- Framework constraint to raw SQL within Flask apps to isolate model-native security knowledge by bypassing ORM protections and testing autonomous secure coding capabilities.
- Integration of autonomous AI-based penetration testing agent (HackingBuddyGPT) alongside Burp Suite for realistic, modern attacker simulation on AI-generated code.
Datasets
- NIST SP 800-63B PDF document — authentication guidelines — public NIST publication
- Code generated on demand from five AI coding models under defined prompts — synthetic/generated dataset
Baselines vs proposed
- Basic Prompt (BP) Google Antigravity: ~11/34 controls implemented vs Reprompting (RP): ~22/34
- Basic Prompt (BP) Cursor IDE: ~9.5/34 vs RP: ~23/34
- Basic Prompt (BP) Copilot Sonnet 4.5: ~13.5/34 vs RP: ~26/34
- Basic Prompt (BP) Copilot Sonnet 4.6: ~13.5/34 vs RP: ~28.5/34
- Basic Prompt (BP) OpenAI Codex: ~9.5/34 vs RP: ~25.5/34
- NIST Prompt (NP) average compliance 20.5-25.5/34 vs RP average compliance 22-30.5/34
- MFA support absent or inconsistent in BP, SP, and NP, present only in RP for some models
Limitations
- Evaluation constrained to Flask framework and raw SQL with SQLite; results may not generalize to other stacks or ORM-based workflows.
- Manual static analysis introduces potential subjectivity and may miss subtle implementation flaws not easily classified as compliant/non-compliant.
- Dynamic testing scenarios, while covering key exploits (brute force, CSRF, session hijack), do not encompass all possible real-world attack vectors or more advanced threat models.
- Prompting experiments limited to four prompt categories; alternative prompting strategies or few-shot examples were not tested.
- Models evaluated represent a snapshot in time; future model improvements could affect security generation quality.
- The reprompting workflow, while effective, introduces latency and complexity which may not be feasible in rapid development or CI/CD pipelines without tooling support.
Open questions / follow-ons
- How do these findings extend to other programming languages, frameworks, and backend stacks beyond Python Flask with raw SQL?
- Can automated integration of large language model self-auditing (reprompting) be incorporated into continuous integration pipelines effectively without human intervention?
- What is the impact of few-shot or chain-of-thought prompt engineering compared to the single-shot and reprompting strategies studied?
- How do adversarially crafted prompts or malicious user inputs influence the security of model-generated code, and can models be hardened against prompt injection attacks?
Why it matters for bot defense
This study highlights key security deficiencies in AI-generated authentication code that are highly relevant to bot defense and CAPTCHA practitioners. The findings emphasize that relying on generated code from LLM assistants without explicit security guidance will likely produce vulnerable systems susceptible to brute-force and session hijacking attacks. Iterative prompt refinement and self-auditing by the model itself are required to realize lasting defense-in-depth implementations.
For CAPTCHA engineers, enforcing secure rate limiting, account lockout, and MFA are critical to preventing automated abuse driven by credential stuffing and brute forcing, as the paper evidences these features are often omitted in naive AI-generated code. Moreover, session security improvements such as HttpOnly and Secure cookie flags and CSRF protections are vital hygiene factors that impact how human and bot traffic can be reliably segregated. This work thus informs CAPTCHA-related backend developers considering LLM-assisted code generation that comprehensive security and bot mitigation require standards-driven continuous verification rather than one-shot prompt engineering.
Cite
@article{arxiv2607_23710,
title={ The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting },
author={ Ishpuneet Singh and Shreyas Mahajan and Gurjot Singh and Maninder Singh },
journal={arXiv preprint arXiv:2607.23710},
year={ 2026 },
url={https://arxiv.org/abs/2607.23710}
}