LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents

Source: arXiv:2606.30697 · Published 2026-06-29 · By Yogeswar Reddy Thota

TL;DR

LUMOS addresses a critical mismatch between current operating systems optimized for human users and the needs of AI agents interacting with desktop and web environments. Human UIs rely on pixels, visual affordances, and layout cues, but AI agents require compact, symbolic, and grounded semantic state plus reliable action primitives. The paper introduces LUMOS, a middleware semantic interaction layer that converts native accessibility metadata (e.g., Microsoft UI Automation trees and browser accessibility structures) into stable machine-readable blueprints. This produces a structured representation of visible UI elements identified by roles, names, states, bounds, and affordances. An LLM planner consumes this blueprint and issues constrained visible UI actions in an observe–plan–act loop, reducing reliance on costly pixel-level perception like screenshots and OCR parsing.

The LUMOS prototype demonstrates feasibility by controlling native Windows apps like Notepad and web browsers using semantic states and universal action schemas, rather than brittle UI scripts or vision-based methods. It enables live semantic pointer grounding to identify and target UI elements under the cursor more efficiently than screenshot cropping plus OCR. The paper argues this layered architecture—preserving visible UI interaction while providing a semantic plane for AI agents—is a practical step toward AI-native operating systems. It reuses existing accessibility infrastructure, avoids backend API bypasses, and offers a more efficient, debuggable, and safer interaction substrate for LLM agents automating desktop tasks.

Key findings

• LUMOS converts native OS accessibility metadata and browser accessibility trees into compact semantic blueprints with stable element IDs, roles, names, values, bounds, and action affordances.
• Live semantic pointer grounding queries the UI element under or near the cursor through OS automation APIs, offering lower latency and higher confidence than vision-based crop-and-OCR pointer interpretation.
• The universal action schema exposes only visible UI primitives such as click, type_text, open_app, and press_key, avoiding hidden backend calls and enabling safer, reversible interactions.
• The LLM planner emits a single JSON action per step from a constrained schema, enforcing an observe–plan–act loop with no hallucinated long sequences.
• LUMOS repaired early typing failure modes in Notepad by rejecting literal instruction copy, converting append to replace text actions, requiring explicit finish actions, and pasting multiline content atomically.
• Blueprint compression experiment plans propose measuring semantic blueprint token sizes versus raw screenshots and OCR transcripts, hypothesizing a significant token reduction.
• The prototype supports Windows native apps and browsers on Windows via UI Automation and DOM extraction, validated with regression tests for action schema and memory correctness.
• Safety policies require explicit confirmation for high-risk operations like sending email or changing system settings, prioritizing visible UI actions as user-equivalent.

Threat model

The threat model is non-adversarial and focuses on reliable grounding and safety of AI agents interacting with operating system UIs. The AI agent cannot bypass visible interfaces or directly call risky backend APIs without explicit user approval. The OS prevents unintended destructive or privileged actions through a combination of visible UI constraints and confirmation policies. Adversary capabilities, such as malicious manipulation of UI semantics or system compromise, are not the focus.

Methodology — deep read

The paper frames the threat and use model as an AI agent interacting with existing human-centric desktop and web UIs through operating system semantic accessibility APIs. The adversary model is not explicitly adversarial but focused on improving agent reliability and grounding.

Data derives from live UI Automation (UIA) trees on Windows desktop apps and browser accessibility/DOM trees. The system extracts UI elements' properties: element ID, role/control type, accessible names, current values, bounding rectangles, window titles or URLs, and focus context. The semantic blueprint is intentionally compact, excluding decorative features.

LUMOS architecture has four main layers: (1) A perception layer queries UIA and browser accessibility APIs to build a normalized semantic blueprint of the current UI state, assigning stable IDs to elements. (2) Live semantic pointer grounding maps the mouse cursor position to a UI element by OS ElementFromPoint-style queries rather than screen crop plus OCR.

(3) A planner layer uses an LLM prompted with the user goal, recent memory, and current blueprint to output a JSON single-step action from a universal schema (observe, open_windows_search, open_app, click, type_text, set_text, press_key, finish). This enforces an observe–plan–act loop, requiring re-observation after each action and preventing hallucinated multi-step instructions.

(4) The memory and repair layer tracks the current text entered, failures, and recent history to avoid repeated actions, enable replace versus append typing, and enforce explicit task completion (finish).

The runtime executes only validated visible UI actions through OS automation APIs, never hidden backend calls. It also enforces safety policies requiring confirmations for destructive operations.

The prototype was developed in Python on Windows, using native UI Automation via Python automation libraries and browser automation for web UI.

A concrete example: opening Notepad triggers a blueprint extraction of 32 UI elements. The LLM is prompted with a blueprint describing a document/edit field with current text “hello.” The LLM outputs a JSON action to type generated text to element A2. The runtime executes the UI automation click and type steps, then re-observes to confirm action completion. Diagnostics track failure patterns such as repeated topics, literal instruction copying, or appending instead of replacing text, which the bridge repairs automatically.

Evaluation is proposed along four axes: (1) visual grounding (screenshot+OCR+LLM) versus semantic blueprint+LLM pipelines measuring latency, token counts, and success rates; (2) blueprint compression vs raw screenshot sizes; (3) semantic pointer latency vs cursor crop+OCR; (4) multi-step tasks across Notepad, Settings, browsers, file explorer, and mail clients.

The paper emphasizes reproducibility via open source code but no formal human-subject or benchmark evaluation yet. The current prototype implements regression tests for schema enforcement, text handling, memory, and safety but leaves large-scale evaluation and complex workflows as future work.

Technical innovations

• A middleware semantic OS interaction layer that extracts and normalizes native accessibility metadata into stable machine-readable UI blueprints for AI agents.
• Live semantic pointer grounding that maps pointer coordinates directly to UI Automation elements, bypassing expensive screenshot-crop and OCR pipelines.
• A constrained, universal visible UI primitive action schema for LLM planners ensuring safe, reversible, human-visible interactions without hidden backend API calls.
• An observe–plan–act loop architecture where LLMs emit a single validated UI action per step, enabling robust grounding and safe multi-step workflows.

Baselines vs proposed

• Vision-screenshot+OCR+LLM vs LUMOS blueprint+LLM: evaluation planned but no numerical results yet; expected improvements in latency, token counts, and recovery steps (Section VIII).
• Semantic pointer latency: proposed to compare ElementFromPoint OS queries against screenshot crop+OCR approaches; no measured data reported.

Limitations

• LUMOS depends heavily on the quality and completeness of accessibility semantics exposed by applications; poorly supported or custom-rendered apps degrade performance.
• Dynamic UIs can change between observation and action, leading to inconsistency and potential failure requiring costly recovery.
• LLM planners may misinterpret semantic blueprints, select incorrect actions, or fail to robustly determine task completion, necessitating multiple recovery turns.
• Current prototype demonstrated mostly on simple desktop text entry and launch tasks; complex workflows (e.g., mail clients, video editors) remain unsolved.
• Security concerns remain with AI-controlled UI layers needing to prevent unintended destructive actions, credential leaks, or privilege escalation.
• No formal benchmark or human-subject evaluation completed yet; validation limited to regression tests and informal case studies.

Open questions / follow-ons

• How do semantic blueprint-based agents scale to highly dynamic or complex multi-application workflows spanning different OS components?
• What improvements in accessibility metadata standards or richer semantic state exposures are needed to enable robust AI-native operating systems?
• How can strong safety guarantees be formally enforced when AI agents control visible UI interactions in security-sensitive contexts?
• What are the tradeoffs between visible UI actions versus backend API access for reliable yet safe autonomous agent operation?

Why it matters for bot defense

For bot-defense and CAPTCHA research engineers, LUMOS offers an insightful perspective on how AI agents could interact with real desktop and web interfaces at a semantic, machine-readable level, rather than relying on brittle visual or coordinate inference. This semantic interaction model reduces token and computational cost for agents by shifting from screenshot-heavy perception to accessibility API-based grounding. It highlights that modern operating systems already expose rich semantic metadata which could be leveraged to improve automated UI interaction fidelity and safety away from pixel-based guesswork.

From a CAPTCHA and bot defense standpoint, understanding this semantic layer suggests new detection and mitigation avenues: interaction patterns grounded in accessible UI semantics might differ from raw visual interaction scripts or bots operating at pixel level. Also, LUMOS shows that future AI-native agent platforms may require accessibility and UI automation APIs to be hardened and audited as security boundaries, as they effectively expose an agent interface plane. Bot-defense practitioners should monitor such developments as they could shift attack surfaces and defense strategies from visual spoofing to semantic-level manipulation or authorization controls.

Cite

@article{arxiv2606_30697,
  title={ LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents },
  author={ Yogeswar Reddy Thota },
  journal={arXiv preprint arXiv:2606.30697},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30697}
}

Read the full paper

• arXiv:2606.30697 (abstract)
• arXiv:2606.30697 (PDF)