From Prompt to Embodied Simulation: Using Generative AI to Create AR Physics Learning Tools
Source: arXiv:2607.24709 · Published 2026-07-27 · By Ofek Levy, Joshua Glazer, Noah D. Finkelstein, Yossi Ben-Zion
TL;DR
This paper presents a novel approach to creating augmented-reality (AR) physics learning tools by leveraging generative AI, specifically large language models (LLMs), to produce hand-controlled AR simulations from structured natural-language prompts. Unlike traditional simulations controlled via sliders or text inputs, these AR tools enable embodied interaction: students manipulate physical quantities using intuitive hand gestures captured by a camera, thus integrating sensorimotor experience into physics learning. The authors demonstrate this methodology through a sine wave and lamp glow simulation controlled by pinch-and-spread finger gestures and extend it to electric charge fields and the right-hand rule in electromagnetism. They pilot-tested the wave-and-lamp simulation with 29 students in an introductory physics course, showing high engagement and positive perceptions of the embodied interface, though without a controlled learning outcome study. The core contribution is lowering technical barriers so teachers and students without coding expertise can rapidly develop customized, interactive AR physics simulations in a browser, using freely available tools like MediaPipe Hands and Three.js, guided entirely by natural language prompts.
Key findings
- A structured natural-language prompt composed of four fixed elements (tools, display, hand controls, optimization) suffices to generate interactive AR physics simulations with hand gesture input.
- Using Gemini 3.1 Pro in Canvas mode, the model can produce runnable HTML code that integrates camera-hand tracking (MediaPipe Hands) and overlay graphics (Canvas or Three.js) in a browser.
- In the wave-and-lamp simulation, vertical thumb-index finger gap controls amplitude (wave height and lamp brightness), horizontal gap controls wavelength (color from violet to red), with smooth transitions and phase-continuous wave animation (Fig 1).
- In a pilot with 29 medical imaging students, 100% agreed the pinch-spread gesture helped them feel the wavelength concept (mean 4.52/5) and amplitude (mean 4.59), with 93% finding the gesture natural and 86% reporting increased engagement over regular learning.
- Iterative refinement of prompts addressing AR-specific challenges (e.g. hand distance normalization, motion smoothing, jitter reduction) yielded stable and aligned simulation behavior in real classroom lighting.
- Variants created with the same prompt structure include 3D electric field visualization around one or two charges (Fig 2a,b) and a non-mirrored right-hand rule visualization aligned to the student's hand (Fig 2c).
- The simulation runs entirely client-side in a single HTML file, requiring no installation, enabling easy deployment and hands-on modification by non-coders during class.
- The approach exposes a practical workflow for embodied cognition in physics education, enabling physical interaction beyond traditional GUI controls.
Methodology — deep read
The paper's methodology centers on transforming natural language prompts into interactive AR physics simulations using generative AI. The threat model is not adversarial but focuses on enabling educators and learners without coding skills to create stable interactive simulations with embodied input. The adversary limitations are not applicable here.
The data inputs consist mainly of device camera feeds capturing hand motions. The critical preprocessing involves using the MediaPipe Hands library to detect finger joints and landmarks in real-time. This detection provides normalized coordinates of finger tips, which serve as parameters to control physics quantities.
Their core architecture combines these components: a client-side HTML5 canvas or Three.js rendering layer overlays drawings on the live camera feed. The physics simulation is minimal and represented visually through parameter-dependent drawings (e.g. sine wave, glowing lamp, 3D arrows). The generative AI produces the entire front-end code — including setup, hand tracking initialization, gesture interpretation, graphics rendering, and UI text.
The prompt is divided into four fixed content elements: which libraries to use and how to present the camera feed; what objects to draw and how they represent physical quantities; how gestures control parameters and which finger movements are tracked; and optimization instructions to reduce jitter, maintain phase continuity, and handle temporary tracking loss.
Training is not part of the method—rather, the authors rely on a commercial LLM (Google Gemini 3.1 Pro) in Canvas mode that can instantly generate runnable code from the prompt. Iteration involves running the generated code, observing errors or instability in hand tracking or rendering, and sending a corrective prompt phrased in natural language. Examples include normalizing the thumb-index gap by palm width and locking wave phase to avoid visual jumps.
Evaluation protocol includes three checks: technical (robust hand tracking under classroom lighting with stable mirroring and smooth responses), physical (parameter mappings consistent with physics, e.g., wavelength-color correspondence), and pedagogical (whether gesture control feels natural and non-distracting). Classroom validation with 29 students involved deploying the wave-and-lamp simulation, gathering survey responses about perceived engagement and intuitiveness rather than quantitative learning gains.
No formal cross-validation or statistical testing is reported beyond descriptive survey statistics. The source code is generated on-demand by the LLM and provided to users as a single HTML file, enabling reproducibility. The datasets are the live camera feeds and the human hand gestures captured during use, which are ephemeral. Supplementary AR simulations further demonstrate prompt reusability across topics.
A concrete example: the user opens the HTML file produced by the LLM, grants camera access, and performs pinch-spread gestures with the right hand. The code uses MediaPipe Hands to detect finger tips, normalizes the gap by palm width, and maps vertical gap size to amplitude controlling sine wave height and lamp brightness, while horizontal gap size sets wavelength which affects wave color and lamp glow color. The wave animates horizontally at a fixed speed, continuously updating with stable phase. When tracking is lost, last parameter values persist. This creates an embodied experience where hand motions directly tune abstract physics parameters represented visually in AR.
Technical innovations
- Using a fixed four-element structured natural language prompt to automatically generate complex, interactive hand-controlled AR physics simulations fully client-side for non-coders.
- Combining real-time MediaPipe Hands finger tracking with dynamic graphical visualizations (Canvas and Three.js) directly over a mirrored live camera feed, maintaining coordinate alignment with sensorimotor gestures.
- Introducing motion-stability optimizations in the prompt (e.g., gap normalization relative to palm width, phase-continuous sine wave animation) to handle noise and preserve smooth embodied control.
- Demonstrating that embodied cognition principles (gesture-based control instead of sliders) can be integrated into physics education through generative AI tools accessible to educators and students.
Baselines vs proposed
- No quantitative baseline comparison reported; evaluation is based on user survey feedback and qualitative validation.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24709.
Fig 1: The wave-and-lamp simulation. The gap between the thumb and index finger controls the amplitude
Fig 2: Three simulations built from the same four-element prompt: (a) a positive charge on the fingertip, with
Limitations
- Pilot study conducted on a small sample (29 students) without a control group or pre-post learning assessment, limiting conclusions about learning efficacy.
- No robustness evaluation under varying lighting conditions, hand sizes, or different camera qualities was reported.
- The physical accuracy of simulations is qualitative rather than quantitative; for instance, the lamp glow mapping wavelength to color is non-physical and simplified.
- Limited discussion of potential accessibility issues for students with motor impairments or limited hand dexterity controlling the pinch gesture.
- The approach depends on commercial LLM availability and the MediaPipe Hands library, which might impose constraints on deployment or domain adaptation.
- No adversarial testing or security considerations around camera data privacy or fake gesture attacks were analyzed.
Open questions / follow-ons
- What is the quantitative impact of embodied AR simulations versus traditional slider-based simulations on measurable physics learning outcomes?
- How do different gesture types and embodied interfaces affect cognitive load and engagement across diverse learner populations?
- Can the generative prompt approach be extended to support multi-user or collaborative AR learning experiences?
- How robust and generalizable are the generated simulations across diverse environments, hand sizes, and camera qualities?
Why it matters for bot defense
This research offers a practical pathway to embedding embodied human-computer interaction within browser-based AR physics simulations generated via large language models. For bot-defense or CAPTCHA engineers, the techniques of gesture-based parameter control and real-time hand tracking may inspire novel challenges for automated bot solvers, as the human sensorimotor dexterity and coordination required to manipulate physical quantities in 3D space could raise the attack cost of bots. At the same time, the work shows how generative AI can dramatically lower barriers to creating complex gesture-controlled interactive content, which could influence CAPTCHA design towards richer, more embodied interaction modalities beyond text or click puzzles. However, no explicit security or adversarial evaluation was performed, so direct application to bot defense would require additional research into robustness against spoofing or replay attacks on camera-based gesture input. Practitioners should weigh the embodied interaction benefits while also investigating the potential vulnerabilities introduced by reliance on computer vision and generative AI code synthesis.
Cite
@article{arxiv2607_24709,
title={ From Prompt to Embodied Simulation: Using Generative AI to Create AR Physics Learning Tools },
author={ Ofek Levy and Joshua Glazer and Noah D. Finkelstein and Yossi Ben-Zion },
journal={arXiv preprint arXiv:2607.24709},
year={ 2026 },
url={https://arxiv.org/abs/2607.24709}
}