Automatic Translation of Unstructured Requirements into Linear Temporal Logic through Large Language Models
Source: arXiv:2608.06287 · Published 2026-08-06 · By Alexandra Newcomb, Omar Ochoa
TL;DR
This paper addresses the challenging task in requirements engineering of automatically translating unstructured natural language (NL) requirements into formal specifications expressed in Linear Temporal Logic (LTL). This is particularly important for safety- and mission-critical systems where rigorous formal verification depends on mathematically precise specifications. The key novelty is an empirical evaluation of six contemporary off-the-shelf Large Language Models (LLMs), including GPT 5.2, Claude Opus 4.5, Gemini, and Llama variants, to generate LTL formulas directly from unstructured NL inputs without task-specific fine-tuning. The study uses a heterogeneous benchmark of 15 structurally diverse requirements, manually formalized for ground truth, with 5 independent LLM outputs per requirement-model pair, totaling 450 candidate LTL formulas. Evaluation combines the pass@k metric, measuring the probability of a correct candidate among the top k samples, and a novel self-consistency measure quantifying syntactic reproducibility of outputs across stochastic trials.
Results demonstrate that current general-purpose LLMs achieve practically significant performance in this NL-to-LTL translation task using only few-shot prompting. For easy and medium difficulty requirements, models achieve very high pass@5 (up to 1.0), near-perfect correctness, while harder requirements see diminished but nontrivial performance (e.g. pass@5 as low as 0.33 for some very hard cases). Self-consistency scores range from 0.37 to 0.67 across models, indicating moderate reproducibility. The paper also emphasizes improving non-expert interpretability by eliciting natural language explanations associated with generated formulas and considering timeline-based LTL visualization as complementary validation aids. The findings suggest modern LLMs are viable semi-automated front-end assistants for formalizing unstructured NL requirements, reducing reliance on costly formal methods expertise and structured requirement templates.
Key findings
- Pass@1 averaged across all models and requirements ranges from 0.17 (hard cases) to 1.00 (easy cases), showing substantial variance by requirement complexity.
- Overall pass@5 across all models averages as high as 0.93 (e.g. GPT 5.2 and Gemini 3 Pro) indicating strong performance given multiple attempts.
- Self-consistency scores vary from 0.37 (GPT 5.2) to 0.67 (Gemini 3 Pro), reflecting partial reproducibility of identical LTL formulas across independent runs.
- Most common translation errors were missing components of the requirement (32 instances), incorrect proposition formulation (29), and incorrect logical implications (23).
- Requirements with clear temporal structure (trigger-response, until patterns) yield higher success rates, whereas those involving role-based exclusivity or subtle implication errors resulted in poor performance.
- Few-shot prompting with only two examples was sufficient to attain strong performance without further model fine-tuning or synthetic training.
- The pairing of LTL formulas with model-generated natural language explanations improved human interpretability and validation of the generated formal specs.
- A small heterogeneous dataset of 15 unstructured NL requirements drawn from multiple industrial and academic sources was sufficient to stress-test model generalization.
Threat model
n/a (this work does not consider adversarial threats or attack capabilities but focuses on improving automated NL-to-LTL formalization reliability).
Methodology — deep read
The study is motivated by the difficulty in translating unstructured natural language requirements into precise formal specifications needed for formal verification of critical software systems. The adversary (or challenge) here is the ambiguity, domain specificity, and syntactic variability of unstructured NL that traditional rule- or template-based pipelines struggle to handle reliably.
The dataset derives from the PURE benchmark, consisting of 15 carefully selected NL requirements representing a breadth of temporal reasoning patterns and covering easy, medium, hard, and very hard translation difficulties. These requirements were originally drawn from seven different Software Requirement Specification documents from both industrial and university contexts, ensuring domain and style heterogeneity. Each requirement was manually formalized into an LTL formula by experts. Because multiple logically equivalent LTL encodings exist for many requirements, the human LTLs serve as guides rather than exact ground truth. An evaluation checklist was created per requirement to describe expected propositions, temporal operators, and logical relations.
Six off-the-shelf large language models were chosen to represent a range of sizes, architectures, and inference cost profiles: OpenAI GPT 5.2, GPT 5-mini; Anthropic Claude Opus 4.5; Google Gemini 2.5 Flash, Gemini 3 Pro Preview; and NVIDIA Llama 3.3 Nemotron Super 49B v1.5. The inclusion of smaller and local-only runnable models like Llama addressed practical deployment and data governance concerns.
Few-shot prompting was employed, supplying two paired NL-to-LTL examples in ASCII operator notation per prompt. The prompt also requested a brief natural language explanation linking parts of the NL requirement to the generated LTL subformula components to enhance output interpretability. Five independent stochastic runs were performed for each Requirement × Model pair (15×6×5 = 450 LTL formulas total), ensuring isolation of stochastic sampling variability with no carryover context.
Evaluation combined manual semantic correctness assessment of each candidate formula in relation to the original NL and the checklist, focusing on trigger identification, temporal scoping, well-formedness, and logical correctness (i.e., formula correctly true/false per requirement scenario). The pass@k metric computed the probability that one of the top k samples is correct, with k ∈ {1,3,5}. A self-consistency score measured the proportion of output pairs considered syntactically equivalent across the five runs per requirement-model pair, based on exact formula structure modulo symbols and parentheses.
Qualitative error analysis cataloged common failure patterns such as missing temporal operators, incorrect implications, and incorrect proposition definitions. Results were tabulated to analyze performance trends by requirement difficulty and model. Additionally, discussion includes the use of ltl2timeline, a visualization tool that converts LTL formulas to timeline diagrams, to aid non-expert understanding and validation of outputs.
The entire pipeline exemplifies an end-to-end experiment evaluating state-of-the-art general-purpose LLMs for zero/few-shot unstructured NL-to-LTL translation, measuring correctness, consistency, and interpretability, without any supervised fine-tuning or synthetic data augmentation. The manual semantic evaluation represents the key ground truth process given complexity of the task.
Technical innovations
- Use of few-shot prompting to generate LTL formulas directly from unstructured, heterogeneous NL requirements without task-specific fine-tuning.
- Quantitative evaluation combining pass@k metrics with a novel self-consistency score to measure syntactic reproducibility of LLM outputs across stochastic trials.
- Incorporation of automatic natural language explanations accompanying generated LTL formulas to improve interpretability and validation for non-expert users.
- Empirical benchmarking across multiple modern, production-quality LLMs spanning different families and scales on a diverse unstructured NL-to-LTL translation task.
Datasets
- PURE dataset benchmark subset — 15 unstructured natural language requirements — public, collected from industrial and academic Software Requirement Specifications
Baselines vs proposed
- GPT 5.2: pass@1 = 0.83 vs GPT 5-mini: pass@1 = 0.63
- GPT 5.2: pass@3 = 0.90 vs Claude Opus 4.5: pass@3 = 0.84
- Gemini 3 Pro Preview: pass@5 = 0.93 vs Llama 3.3 Nemotron: pass@5 = 0.93
- Self-consistency: Gemini 3 Pro Preview = 0.67 vs GPT 5.2 = 0.37
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06287.
Fig 1: Summary of experimental setup.
Limitations
- Small benchmark dataset of only 15 requirements limits statistical power and domain coverage.
- Manual semantic evaluation, though thorough, may introduce subjectivity given multiple acceptable LTL encodings.
- No adversarial evaluation or robustness testing against deliberately ambiguous or conflicting NL inputs.
- Evaluation restricted to average pass@k over small number of stochastic samples per prompt, potentially missing rare correct solutions.
- No formal comparison to fine-tuned or supervised NL-to-LTL models or symbolic NLP pipelines controlling for data size.
- Some important failure modes (e.g., subtle implication reversal in Req 6) highlight limits in LLM understanding of logical nuance.
Open questions / follow-ons
- How would supervised fine-tuning or synthetic data augmentation improve the translation accuracy and consistency of LLMs on unstructured NL-to-LTL tasks?
- Can interactive, human-in-the-loop refinement workflows effectively leverage LLM-generated explanations and partial formulas for scalable semi-automated formalization?
- What are the effects of incorporating domain-specific knowledge or terminology into prompts or model pretraining to tackle role-sensitive and permission-based requirements?
- How robust are these translation methods under adversarially ambiguous, conflicting, or incomplete natural language input requirements?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, the paper offers insights into the capabilities and limitations of state-of-the-art LLMs to translate complex, ambiguous natural language specifications into precise formal logic representations without heavy domain-specific engineering. This is relevant where formal constraints must be defined for system behavior under various inputs, including automated challenge-response protocols or verification of rule correctness. The demonstrated combination of machine-generated logical formulas together with natural language explanations and timeline visualizations suggests practical workflows to semi-automate translating informal security policies or CAPTCHA protocols into formal temporal logic suitable for automated reasoning or model checking. However, notable failure modes involving subtle implication errors and incomplete logical conditions caution that fully automated solutions still require human oversight, especially for nuanced or role-sensitive bot-defense rules. Moreover, the study highlights the value of multiple samples, consistency measurements, and explainability in deploying LLM-based formalization tools reliably in security-sensitive applications. Overall, this represents a promising step toward reducing the expertise barrier in defining formal CAPTCHA and bot-defense policies, enabling faster iteration and validation without requiring deep logic specification expertise.
Cite
@article{arxiv2608_06287,
title={ Automatic Translation of Unstructured Requirements into Linear Temporal Logic through Large Language Models },
author={ Alexandra Newcomb and Omar Ochoa },
journal={arXiv preprint arXiv:2608.06287},
year={ 2026 },
url={https://arxiv.org/abs/2608.06287}
}