CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts

Source: arXiv:2607.21394 · Published 2026-07-23 · By Runze Cai, Yuxuan Huang, Lin-Ping Yuan, Kexin Xiang, David Hsu, Collier Nogues et al.

TL;DR

This paper presents CRAFT, a novel approach to support fiction writers using wearable AI smart glasses that facilitate transforming real-world observations into fictional narrative elements in situ. It addresses a key gap in creative writing tools that do not capture or assist with moments of inspiration as writers encounter them in daily life. Through three sequential studies involving interviews with 9 writers, co-design workshops with 16 participants, and 24 field trial sessions with 8 writers, the authors explore the desirability, feasibility, and real-world potential of CRAFT’s context-aware, mixed-initiative AI to augment perception, promote authenticity, and preserve writer agency. The system uses a multimodal large language model that analyzes the wearer’s environment, context, and story to suggest creative sparks, transform user speech and photos into fictional content, and maintain immersion via AR overlays. The field trials reveal benefits such as enriched, authentic ideas from serendipitous encounters, micro-creation enabling writing in brief moments, and embodied character perspectives grounded in physical context. The authors contribute design goals and interaction mechanisms foundational for future human-AI collaboration tools in creative writing on wearable platforms.

Key findings

• Interviewed writers identified an “authenticity triad” essential for believable fiction: factual accuracy, logical/behavioral consistency, and emotional/psychological authenticity.
• Writers face key barriers in situ: time constraints, memory fade from fragmented notes, and a mismatch between rapid thought and slower typing output.
• All interviewed writers desired wearable AI functions including augmenting perception to bridge reality-fiction gaps and direct reality-fiction transformation for faster iteration.
• Writers emphasized preserving creative agency and enjoyment, and maintaining life-art boundaries to avoid exhaustion and trauma triggers.
• The technology probe on smart glasses leverages a multimodal LLM to continuously analyze environment and story context for proactive creative suggestions on the wearer’s AR display.
• Supported field trials across 24 sessions with 8 writers yielded 8 fiction stories and revealed emergent usage patterns like micro-creation and serendipitous inspiration from daily encounters.
• Writers reported enriched and authentic fictional ideas grounded in their physical surroundings improved by the wearable AI’s in-situ assistance.
• Design goals for wearable creative AI include bridging perception gaps, promoting authenticity triad while allowing fictionalization, and preserving authorial control and enjoyment.

Threat model

n/a (the paper investigates user-centered wearable AI for fiction writing and does not target adversarial threat scenarios).

Methodology — deep read

The research is structured into three progressive studies following Design Thinking principles, emphasizing desirability, feasibility, and viability.

1. Threat Model & Assumptions: The adversary in a security sense is not addressed explicitly since this is an HCI creative writing study; instead, the study assumes typical users are creative writers using smart glasses for fiction-authored content transformation with an AI assistant.

2. Data Collection: The authors collected qualitative and usage data via three studies: (a) semi-structured interviews with 9 diverse fiction writers varying in experience and AI familiarity to understand current practices and needs; (b) co-design workshops with 16 participants, including writers and researchers, using an initial technology probe to ideate concrete interaction mechanisms; and (c) supported field trials where 8 writers used a refined version of the technology probe across 24 sessions in real-world environments (including visits to landmarks) over multiple days, producing 8 fiction stories.

3. Architecture & Algorithm: The CRAFT system is implemented as a technology probe on AR smart glasses with a mixed-initiative interaction model. The probe captures multimodal input: first-person video, audio speech, timestamp, and location. It employs a multimodal large language model (MLLM) that continuously analyzes environmental context (objects, atmosphere, people) and story context initialized by user input (voice for theme and style). The MLLM detects semantic associations between the real environment and the fictional world, generating proactive creative suggestions displayed as glanceable overlays through the glasses. Authors can respond with voice or a ring mouse click to transform observations into fictional snippets. The system supports functions such as plot construction, Q&A, role-playing characters, and re-observing transformed scenes with partial reality-fiction blending.

4.Training Regime: The paper does not provide specific LLM training details; presumably, existing pretrained LLMs are adapted or queried in real time. The focus is on interaction design rather than training novel AI models.

5. Evaluation Protocol: Evaluation comprises qualitative thematic analysis of writer interviews, co-design feedback, and rich longitudinal field trial observations centered on usage patterns, creative practice emergence, and authorial perceptions of benefits and challenges. The trials included multi-session use in varied locations, capturing naturalistic writer-AI collaboration. Success metrics are based on user-reported authenticity, creative agency preservation, and idea enrichment rather than quantitative benchmarks.

6. Reproducibility: The study focuses on design explorations rather than releasing code or datasets. The technology probe is described conceptually with illustrative UI, but code and detailed AI model implementations are not released publicly. The interpretive results rely on qualitative user data and thematic analyses.

A concrete example from field trials: A novelist observes the Arc de Triomphe. The system proactively suggests a creative spark linking the arc’s shape to a portal in her fantasy plot. She clicks the ring mouse and verbally ideates. The AI transforms her narration and scene into fictional content displayed as an AR overlay blending real and fictional elements while she continues story development in situ.

Technical innovations

• Integration of multimodal real-world environmental sensing with large language models to generate proactive, context-relevant narrative suggestions for fiction writing via smart glasses.
• Mixed-initiative interaction combining AI proactive creativity support with user control through voice and ring mouse input to enable in-situ reality-fiction transformation.
• Design goals and mechanisms explicitly balancing augmenting reality perception, promoting a layered authenticity triad, and preserving writer creative agency and emotional well-being.
• Operationalization of wearable creative AI as a technology probe that supports seamless observe→suggest→express→transform→re-observe loops to maintain immersion in physical and fictional worlds concurrently.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21394.

Fig 1: Top: Novelists often draw inspiration from daily life (e.g., observing a landmark) but face cognitive gaps (e.g., memory

Fig 2: Relationship Between RQs, Studies, Main Results, and Usage of Technology Probe.

Fig 3: CRAFT probe overview. Smart glasses capture first-person video, speech, time, and location. Using mixed-initiative

Fig 4: Design workshop procedure: (1) technology probe trial, (2) design ideation with a simulated AR tool where users can

Fig 5: Workshop examples of Reality-Fiction Transformation. Real-world entities (top) map to fictional elements—setting,

Fig 6: Some locations for Study 3, selected based on participants’ preferences and location availability.

Fig 7: Examples of Transformed Half-Real/Half-Fiction Visualizations in Study 3.

Fig 8: Usage Patterns of CRAFT in Study 3: (a) interaction count percentage for all participants. Note: Global Content refers

Limitations

• Small sample sizes in interviews (9 writers), co-design workshops (16 participants), and field trials (8 writers) limit generalizability.
• Qualitative, exploratory nature without formal quantitative or adversarial evaluation; lacks cross-validation or distribution shift testing.
• No released source code or LLM model specifics impede direct reproducibility or benchmarking.
• The system depends on state-of-the-art multimodal LLM capabilities whose performance and biases may vary and are not extensively evaluated.
• Long-term sustainability and integration into varied writing workflows remain untested beyond modest field trials.
• Potential cognitive load or distraction effects of constant AI suggestions in daily life not rigorously measured.

Open questions / follow-ons

• How can multimodal AI models be further improved to capture and transform subtle emotional and contextual cues for richer fictionalization?
• What are best practices for balancing AI proactivity with writer control to maximize creativity without undermining agency or causing distraction?
• How can such wearable creative AI systems scale across diverse genres, languages, and user expertise levels?
• What long-term effects on writers’ creativity, workflow, and emotional well-being arise from extended use of in-situ AI wearable assistants?

Why it matters for bot defense

Though this research focuses on wearable AI for creative writing rather than security, its insights into context-aware, multimodal interaction, mixed-initiative control, and preserving user agency have conceptual relevance to CAPTCHA and bot-defense engineers. The deliberate balance between proactive AI suggestion and human agency underscores challenges in designing collaborative AI that aids complex cognitive tasks without overwhelming or misguiding users. Moreover, the work exemplifies rich environmental context sensing combined with AI reasoning in a wearable form factor, which can inspire novel approaches to human verification that dynamically adapt to real-world context. The authors’ emphasis on minimizing cognitive burden and emotional impact also aligns with usability goals critical to CAPTCHA experience. Practitioners might consider how multimodal sensing and AI-driven context interpretation could improve detection of bots or automated abuse through behavioral and situational cues captured via devices. However, direct technical crossover is limited since the paper does not address adversarial or security threat models explicitly.

Cite

@article{arxiv2607_21394,
  title={ CRAFT: Exploring Wearable Creative AI on Smart Glasses for Fiction Writing in Real-World Contexts },
  author={ Runze Cai and Yuxuan Huang and Lin-Ping Yuan and Kexin Xiang and David Hsu and Collier Nogues and Jussi Holopainen and Shengdong Zhao },
  journal={arXiv preprint arXiv:2607.21394},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21394}
}

Read the full paper

• arXiv:2607.21394 (abstract)
• arXiv:2607.21394 (PDF)