TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning

Source: arXiv:2607.26307 · Published 2026-07-28 · By Rwaida Alssadi, Muntaser Syed, Balaji Kasula, Lamine Deen, Majed Alotaibi, Mohammed Alghamdi et al.

TL;DR

TraceCoder addresses a major opacity problem in LLM-based code generation agents: the lack of fine-grained, auditable traceability of code evolution during iterative benchmark-driven repair. Contemporary systems treat generated code as a black box with no causal record linking lines to test failures or repair rationale. TraceCoder introduces a novel approach combining a relational snippet-history database that captures every code snippet's repair events with explicit benchmark failures and LLM explanations, a browser-based visualization tool for interactive provenance browsing, and a fractional position-key indexing scheme enabling stable snippet identifiers that preserve lexicographic order despite edits.

Evaluated on 30 algorithmic programming tasks across two provider pairs, TraceCoder demonstrates rich provenance capture: up to 30% of code snippets carry traceable repair-event history, with an average of 1.14 repair rounds per changed snippet. The system enables detailed causal narratives explaining which failing benchmarks triggered specific code changes. A visual heatmap viewer highlights change intensity line-by-line, supporting auditing and debugging. Case studies on Fibonacci, expression evaluation, and Roman numerals illustrate how progressive rounds of bug exposure and repair are recorded and replayed. This work is a step toward explainable, accountable automated code generation suitable for regulated production environments.

Key findings

• TraceCoder records, on average, 20.7% (Gemini 2.0 Flash) to 30.0% (Grok-3-beta + DeepSeek) of code snippets with one or more snippet_failure rows, providing fine-grained provenance.
• Mean snippet_failure rows per changed snippet is approximately 0.96 (Gemini) to 1.14 (Grok+DeepSeek), reflecting multi-round iterative repairs.
• In 10/30 tasks, the maximum iteration budget of 6 rounds was exhausted, highlighting the challenge of subtle edge-case bugs.
• Database provenance size remains modest (~47 KB Gemini, ~54 KB Grok+DeepSeek) despite verbose failure texts, keeping query latency low.
• The fractional indexing scheme ensures lexicographically ordered, stable snippet keys that support insert/delete/modify without costly rebalancing or key rewrites.
• Convergence iterations vary widely; some tasks solve in 1-4 iterations, others reach the max 6 iterations for complex edge cases.
• Benchmark failure explanations and repair rationale are stored verbatim per snippet per round, enabling causal audit queries with simple SQL selects.
• Case studies demonstrate the ability to reconstruct the full code evolution narrative line-by-line, including benchmark that triggered each change.

Threat model

The system implicitly considers an adversary or threat scenario where lack of explainability and auditability in black-box LLM code generation impairs trust and compliance. The adversary is any actor who would exploit or introduce undetected bugs or malicious code in automatically generated programs without provenance. The defense is to establish traceability by recording causal links between benchmarks that fail and the incremental code changes fixing them. It does not address active tampering with the provenance store or malicious LLMs, but facilitates post-hoc auditing and root cause analysis.

Methodology — deep read

Threat model and assumptions: The adversary is not explicitly defined, but implicitly the system assumes auditing and provenance are necessary to trust automated code generation, treating the LLM and benchmarks as black boxes whose internal reasoning is unobservable. The goal is to maintain a transparent, tamper-evident causal record linking code changes to benchmark failures.

Data: The authors evaluate on 30 algorithmic programming tasks in two batches. Batch 1: 20 tasks covering mathematical sequences, string processing, algorithms, and numerical/matrix operations. Batch 2: 10 simpler tasks (arithmetic, lists). Two LLM providers used: Gemini 2.0 Flash alone; and a Grok-3-beta (code) + DeepSeek-V3 (benchmark creation) pair. Max iterations per task is 6. Benchmarks are automatically generated by an LLM observing current code to create tests for uncovered edge cases.

Architecture and algorithm: The TraceCoder system comprises four layers: a persistent SQLite-based versioned snippet store; an agent core driving an iterative generate–benchmark–test–fix loop; multi-backend LLM abstraction; and a browser-based explainability viewer rendering annotated source code. The core innovation is a relational schema capturing each snippet version with position-key snippet identifiers and round-tagged provenance rows linking code snippets to benchmark failure metadata and LLM explanations. The fractionally-indexed position-key scheme assigns lexicographically ordered, stable string keys to code snippets enabling arbitrarily fine-grained insert/delete/modify without disrupting adjacent lines. The iterative agent loop issues three distinct LLM API calls per round: generating initial code, creating benchmarks, and fixing failing code snippets, each yielding JSON containing code, explanations, and benchmark descriptions.

Training regime: Not applicable since this is not training a model but rather orchestrating LLM calls in an iterative feedback loop with versioned storage. Multiple rounds of generate-fix cycles are executed until all benchmarks pass or iteration budget is exhausted.

Evaluation protocol: Metrics measured include iteration count until convergence, total number of benchmarks created, fraction of snippets with ≥1 recorded repair event (Chg%), mean number of snippet_failure rows per changed snippet (Avg Rds), and database storage size. Results are compared across two provider configurations. Case studies illustrate causal audit scenarios. No mention of formal statistical testing or cross-validation; held-out tasks represent distribution shift.

Reproducibility: Code release or data is not explicitly mentioned; datasets are public algorithmic tasks but code and benchmark generation rely on commercial and research LLM APIs with proprietary weights. All source code and database schema details are described comprehensively for potential re-implementation.

Concrete example walkthrough: The Fibonacci task (Case Study 1) initializes code producing an off-by-one error on input zero. Iterative benchmarks discover this edge case on round 2, triggering a repair that adds a guard clause for negative input in round 3. Each modified snippet is inserted as a new database row with a fresh position-key and round-tag linkage to the failing benchmark and LLM explanation. The viewer displays line-level heatmaps indicating which lines evolved in which rounds, and auditors can query the database using SQL to trace the causal lineage ('why does this line exist?') to specific benchmark failures. The final code and provenance form a persistent, replayable narrative of the entire repair process.

Technical innovations

• A relational snippet-history schema that permanently links each edited code snippet to the specific benchmark failure and LLM explanation that motivated that change, enabling full causal provenance queries at snippet granularity.
• A novel fractional position-key versioning system (FIS) with tree-node delimiters that assigns stable lexicographically ordered string keys to code snippets, allowing insert, delete, and move operations without rebalancing or key rewrites.
• An iterative benchmark-driven code repair loop architecture where distinct LLM API calls generate code, create benchmarks, and fix failing snippets, with all intermediate states persistently recorded.
• A browser-based visualization tool rendering the provenance store as heat-mapped, hover-annotated source code enabling interactive explainability and auditing of the code generation narrative.

Datasets

• Algorithmic benchmark suite — 30 tasks — Public tasks from domains of string processing, mathematical computation, data structures (publicly defined tasks but code generation relies on proprietary LLM APIs)

Baselines vs proposed

• Gemini 2.0 Flash (sole provider): mean iterations = 5.45, mean benchmarks created = 4.55, mean snippet churn (Chg%) = 20.7, mean DB size = 46.8 KB
• Grok-3-beta + DeepSeek-V3 (coding + benchmark creation): mean iterations = 2.07, mean benchmarks created = 8.07, mean snippet churn (Chg%) = 30.0, mean DB size = 54.2 KB (excluding one outlier)
• Comparison shows Grok+DeepSeek generates richer benchmark suites and provenance histories, converges faster (fewer fix iterations), and edits a larger fraction of snippets on average.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.26307.

Fig 2: Core schema of the TraceCoder persistent store. History is normalised into

Limitations

• No formal adversarial or robustness evaluation to test tampering or poisoning of provenance records.
• Reliance on external LLM APIs (Gemini, Grok, DeepSeek) with proprietary models limits reproducibility and introduces variability.
• Benchmark correctness issues occur; some generated benchmarks contain incorrect expected outputs, requiring secondary verification.
• Build fragility observed: some repair iterations introduce syntax errors treated as special benchmark failures, indicating brittleness.
• Dataset limited to synthetic algorithmic tasks; no evaluation on large-scale, real-world codebases or industrial code generation workflows.
• Visualization prototype demonstrates feasibility but remains a research tool; scalability and UI usability for large projects not evaluated.

Open questions / follow-ons

• How robust is the fractional indexing and provenance scheme against adversarial tampering or corrupted benchmark data?
• Can TraceCoder be extended to support parallel or distributed iterative repair loops with conflict resolution?
• What is the scalability and performance when applied to large real-world codebases with hundreds of files and thousands of snippets?
• How effective is TraceCoder in detecting and explaining security-critical bugs or vulnerabilities introduced during automated code generation?

Why it matters for bot defense

TraceCoder's approach to explainable and auditable code generation via snippet-level provenance and causal linking between test failures and code fixes offers a novel paradigm applicable to bot-defense and CAPTCHA systems where security-critical logic evolves iteratively through automated agents. Bot-defense engineers could adopt similar provenance tracking and visualization mechanisms to increase trust and diagnosis capabilities of self-updating protection scripts or automated challenge generators, thereby mitigating risks from opaque AI-driven logic changes.

Furthermore, TraceCoder's fractional indexing scheme and snippet-level audit trail could inspire analogous designs for detecting subtle adversarial inputs or regressions in CAPTCHA challenge-response systems over multiple development iterations. While this work is targeted at general code synthesis, the principles of fine-grained causality and persistent reproducible narratives align with transparency and accountability goals essential in bot defense automation.

Cite

@article{arxiv2607_26307,
  title={ TraceCoder: Explainable and Auditable Code Generation with Position-Key Snippet Versioning },
  author={ Rwaida Alssadi and Muntaser Syed and Balaji Kasula and Lamine Deen and Majed Alotaibi and Mohammed Alghamdi and Tyler Ton and Ali Alqarni and Marius Silaghi },
  journal={arXiv preprint arXiv:2607.26307},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.26307}
}

Read the full paper

• arXiv:2607.26307 (abstract)
• arXiv:2607.26307 (PDF)