Thinkink: 2D Spatial Ink-native Interaction with LLMs

Source: arXiv:2607.21468 · Published 2026-07-23 · By Mohammad Hasan Payandeh, Daniel Vogel, Jian Zhao

TL;DR

This paper addresses the gap between traditional 2D ink-based ideation workflows and the integration of large language models (LLMs) for more fluid, spatially grounded human-AI interaction. The authors introduce Thinkink, a novel digital tool that allows users to write and sketch handwritten prompts and receive LLM-generated ink-like responses directly on the shared 2D canvas, preserving the natural spatial and visual aspects of ideation workflows. The approach is guided by a semantic tree representation that interprets ink strokes as drawings, concepts, and requests, enabling context-aware LLM engagement. A lightweight UI with distinct modes controls interaction, mitigating confusion common in mixed-ink/AI systems. The iterative development includes a formative study (N=12) to understand ideation practices, a diagnostic study with a technical probe (N=6) to reveal design challenges, and a final usage study (N=10) to validate how participants incorporate Thinkink in ideation.

Thinkink demonstrates how LLMs can be tightly integrated within an ink-native environment, generating exploratory outputs such as reflective questions and suggestions that support abductive thinking. The system's semantic tree enables LLMs to ground responses in evolving spatial ink semantics rather than isolated text, supporting non-linear idea development. However, user studies identified UI issues like unclear mode boundaries, lack of undo, and misalignment between response formats and user needs, which the authors addressed by explicitly separating note-taking and AI interaction modes in Thinkink. Overall, the work provides both empirical insights to inform future ink-native human-LLM interfaces and a functional prototype that blends pen input with AI as a contextual ideation partner on a fluid 2D canvas.

Key findings

• Formative study (N=12) revealed non-linear, evolving externalization workflows where users start with partial text or rough sketches and continuously revise and reorganize ideas spatially.
• Participants strongly preferred a minimal, paper-like digital inking experience preserving immediacy and low distraction with full-screen pen input and lightweight touch manipulation.
• In the diagnostic study (N=6), users valued the semantic tree-based AI insights that generated exploratory, Socratic questions rather than direct answers, facilitating idea expansion.
• Key interaction challenges included confusion over when the system was interpreting ink vs. note-taking, difficulty undoing actions, opaque LLM context selection, and limited user control over AI engagement.
• Users desired explicit modes to separate note-taking from AI prompting, more direct query capability, and richer LLM responses including visual sketches or diagrams rather than only text.
• The semantic tree model segments ink into drawing nodes, concept nodes, request nodes, and generation nodes, enabling layered LLM interpretation and responses grounded in spatial context.
• Thinkink’s final design features a state machine UI with distinct modes for prompts, insights, request iteration, and generation inspection to clarify human-AI interaction boundaries.
• Final study (N=10) identified four usage patterns validating the Thinkink approach, showing that users integrated the tool into ideation by collaboratively developing ink-based ideas with AI contributions directly on the canvas.

Threat model

n/a. The research focuses on human-AI interaction and usability in a creative ideation context, not on adversarial threat models or security.

Methodology — deep read

1. Threat Model & Assumptions: The authors assume typical ideation users (researchers, designers, technical experts) using pen and paper or digital ink tools to externalize ideas. The adversarial aspect is minimal; the paper focuses on human-LLM collaboration instead of security threats. They implicitly assume the LLM behaves as a context-aware assistant providing exploratory support without malicious intent.

2. Data: Three user studies were conducted. The formative focus group had 12 HCI researchers and students using mixed media for ideation tasks. The diagnostic study involved 6 participants from diverse technical backgrounds using a technical probe on an iPad Pro for two 10-minute ideation tasks. The final usage study included 10 participants using the refined Thinkink prototype. Researchers collected ink artifacts (sketches, notes), screen and interaction logs, audio recordings, and survey data. Data included handwritten text, sketches, concepts, requests, and LLM-generated ink annotations.

3. Architecture / Algorithm: Thinkink’s core innovation is a semantic tree that organizes the 2D canvas ink into a layered structure: drawing nodes (handwritten/sketched strokes), concept nodes (higher-level groupings), request nodes (places where the system identifies opportunity for support), and generation nodes (LLM responses, either spontaneous or in reply to requests). The tree enables the system to ground LLM prompting in spatially and hierarchically related ink content. The interface supports pen and touch input on a large infinite canvas with minimal UI chrome. Users can write, sketch, erase, pan, zoom, and interact via a state machine that explicitly separates note-taking from AI prompting and response inspection modes. The LLM used is Gemini 3 Flash Preview.

4. Training Regime: The LLM model is an external API (Gemini 3 FP), so no model training details are reported. The system manages prompting dynamically by extracting context from the semantic tree, including parent, sibling, and related nodes, for each generation request.

5. Evaluation Protocol: User studies combined qualitative methods (interviews, think-aloud, thematic analysis) with quantitative Likert surveys, usage logs, and artifact inspection. The formative study gathered workflow insights; the diagnostic study identified usability and interaction challenges with the initial probe; the final study observed actual usage patterns with Thinkink. No adversarial or performance benchmarks were done. The main metrics were user-reported experiences and usability feedback.

6. Reproducibility: The paper does not explicitly mention releasing source code or datasets. The LLM component relies on a closed Gemini 3 model by Google. Thus, full replication may require access to the underlying models. However, detailed prompts and UI designs are documented in appendices, supporting independent implementation.

Example Workflow End-to-End: A user begins by writing a high-level ideation goal and rough sketches on the blank canvas. The system, after a short idle, parses the ink into the semantic tree with drawing and concept nodes. It identifies areas where it can offer exploratory questions or alternative directions (request nodes). These appear as semi-transparent ink annotations near the relevant sketches. The user taps them to finalize helpful AI-generated notes, incorporating them into the canvas. In subsequent iterations, the user may explicitly switch modes to prompt the AI with queries, and the system responds with ink-like answers spatially related to the prompt. Throughout, the user can move, erase, or add ink, and the semantic tree updates to maintain AI contextual grounding.

Technical innovations

• A semantic tree structure that organizes ink input into drawing, concept, request, and generation nodes to enable contextually grounded LLM prompting within a 2D spatial canvas.
• Ink-native LLM interaction where user prompts and system responses coexist as ink-like handwritten text and sketches spatially integrated on the same canvas.
• A state machine UI design that explicitly separates note-taking from AI assistance modes, reducing interaction ambiguity and improving user control.
• Exploratory, question-first LLM outputs designed to expand ideation space without constraining user agency, rather than delivering direct answers.

Datasets

• User inking and ideation artifacts from formative study — 15 sketches — private university lab participants
• Ink and interaction logs from diagnostic study — 6 participants — collected with technical probe on iPad Pro
• Ink and interaction logs from final usage study — 10 participants — using Thinkink prototype

Baselines vs proposed

• Technical probe baseline (no explicit modes): 11-item Likert usability mean rating below final Thinkink improved interface (specific numbers not reported)
• Participant preference for exploratory question-style LLM outputs vs. direct answers: 5 of 6 diagnostic participants favored questions
• Boundaries clarity: users reported confusion without explicit modes in technical probe; explicit state-machine UI improved clarity in final prototype (qualitative)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21468.

Fig 1: Example Thinkink usage scenario for logo ideation: (a) user writes high-level ideation goal; (b) system in ‘AI insights’

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Small sample sizes in all studies (12, 6, 10) limit generalizability and statistical power.
• No adversarial evaluation or testing with maliciously crafted ink inputs or LLM manipulation attacks.
• No quantitative performance benchmarks or real-time responsiveness evaluations of the system.
• LLM dependence on a closed proprietary Gemini 3 Flash Preview model limits reproducibility and tuning.
• User studies primarily conducted in controlled lab contexts with university participants; real-world usage and scaling remain untested.
• The system currently generates mostly text-based annotations; limited direct LLM-generated sketching or diagramming was implemented.

Open questions / follow-ons

• How to incorporate richer multimodal LLM outputs such as direct sketch or diagram generation integrated in ink-native fashion?
• How does Thinkink scale for larger, longer ideation sessions involving multiple users and extensive canvases?
• Can the semantic tree and prompting approach adapt to other LLM models or open-source alternatives with comparable efficacy?
• What interfaces and interaction models best balance user agency and AI assistance in collaborative ideation to prevent automation bias?

Why it matters for bot defense

While Thinkink is not directly related to bot defense or CAPTCHA schemes, its technical insights are valuable for developers exploring pen-based, ink-native human-AI interaction paradigms. The semantic tree approach to interpreting mixed ink inputs and grounding LLM prompts in spatial context may inspire novel continuous challenge-response mechanisms that integrate handwriting or sketching as authentication factors. Additionally, the design lessons on explicit interaction modes to reduce user confusion and increase control can inform the user experience around bot detection challenges relying on dynamic pen input. Finally, Thinkink's emphasis on exploratory, non-declarative AI outputs offers a contrast to task-focused AI automation, underscoring the importance of preserving user control and agency in complex AI-assisted user interfaces.

Cite

@article{arxiv2607_21468,
  title={ Thinkink: 2D Spatial Ink-native Interaction with LLMs },
  author={ Mohammad Hasan Payandeh and Daniel Vogel and Jian Zhao },
  journal={arXiv preprint arXiv:2607.21468},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21468}
}

Read the full paper

• arXiv:2607.21468 (abstract)
• arXiv:2607.21468 (PDF)