Beliefs and Misconceptions around Integrated Conversational AI
Source: arXiv:2605.14849 · Published 2026-05-14 · By William Seymour, Adam Jenkins, Mark Cote, Jose Such
TL;DR
This study investigates how users understand and develop trust in integrated Large Language Model (LLM) conversational AI (CAI) features embedded within existing workflows, specifically Microsoft Edge's Copilot. As LLMs become commodified and lose visible markers of origin, users form beliefs rooted in prior mental models of search engines and AI. By conducting semi-structured interviews with 20 participants performing information retrieval and planning tasks using Copilot, the authors observed that users rely on diverse and often conflicting prompting strategies shaped through trial-and-error and social media, rather than official guidance. Participants generally conceptualized Copilot as a sophisticated search engine that summarizes web results and fact-check by consulting the same sources, leading to over-trust reinforced by citations presented with responses. Although users were cautious about potential bias, advertising influence, and inaccuracy, citations greatly increased perceived credibility, often obviating the need for further verification. The study highlights a fragmented understanding of CAI functionality, over-reliance on partial signals of trustworthiness, and the challenges faced by users in adapting to integrated conversational AI in everyday contexts.
Key findings
- Participants employed diverse prompting strategies (e.g., detailed prompts, start broad then refine, concise direct questions) driven by personal trial-and-error rather than vendor guidance (Table 2).
- Most participants conceptualized Copilot as a search engine summarizing web results, often believing popularity metrics (clicks, ranking) influenced answer selection.
- Inclusion of citations significantly increased participant trust in Copilot’s answers, often stopping them from further fact-checking (‘I don’t want to spend more time on it’) (Section 4.3).
- Participants frequently used the same web search engines they believed Copilot used to verify AI-generated answers, raising questions about independence of verification.
- Users displayed awareness of potential bias from sponsored content or Microsoft-link prioritization, but treated in-house source inclusion neutrally as typical business practice.
- Participants showed awareness of hallucinations and accuracy issues, conceptualizing the need to fact-check outputs with authoritative sources, but lacked consistent heuristics to do so.
- Despite understanding AI’s presence in Copilot, participants had vague or inconsistent mental models of AI’s workings and other possible platform applications.
- Privacy concerns were minimal among participants, who largely prioritized convenience over data risks, except for potential misuse in advertising.
Threat model
The study addresses a non-adversarial scenario focusing on ordinary users as 'adversaries' of trustworthiness—that is, their misconceptions and beliefs when interacting with integrated CAI. The adversary is informational uncertainty and over-reliance rather than a malicious entity. Participants are assumed to have no direct access to internal CAI mechanisms and rely on surface heuristics to judge trust.
Methodology — deep read
Threat Model & Assumptions: The adversary here is not a traditional malicious attacker but rather a study of ordinary users' mental models and perceptions when interacting with integrated conversational AI. Participants had prior experience with conversational AI but not Copilot specifically. The study assumes users lack formal training in AI prompting and rely on existing mental models shaped by web search and social media.
Data: The data consists of semi-structured interview transcripts from 20 participants. Participants were recruited through a university's temporary work platform, compensated £20, ages 19-51 (avg 25.3), mostly women (17). They had varied prior CAI exposure (ChatGPT, Siri, Bard, etc.). Two tasks were used: (a) find health benefits/drawbacks of a cooking ingredient and (b) plan a weekend city visit including transport, museum, and restaurant. Task chat logs and verbalized think-aloud protocols were recorded.
Architecture/Algorithm: Not applicable; this is a qualitative HCI study. The focus is on user interaction with Microsoft Edge’s integrated Copilot chat, which uses retrieval-augmented generation grounded in Bing search results.
Training Regime: N/A.
Evaluation Protocol: The primary evaluation consisted of qualitative thematic analysis following Braun and Clarke’s methodology. Interviews were recorded, transcribed, and iteratively coded by two researchers who developed thematic codes collaboratively. No inter-rater reliability scores were reported, emphasizing transferability rather than generalizability. Participant think-aloud data and follow-up questions on trust, reliability, and privacy constructs were analyzed. Quotes illustrate belief patterns. The study focused on exploratory insight rather than quantitative metrics.
Reproducibility: Code, chat logs, or raw transcripts are not publicly released. The Copilot integration relies on proprietary Microsoft technologies, limiting direct replication. However, the interview protocol and analytic approach are described in sufficient detail to replicate a similar study.
Concrete Example: A participant was asked to find health benefits/drawbacks of thyme using Copilot. They issued prompts and observed AI’s summarized answers with citations. On inspecting the linked citations, one was from NHS Foundation Trust (considered reliable) but another was off-topic (mnemonics for conversation with distressed persons). The participant then wrestled with whether to trust the source based on relevance and URL type, illustrating the nuanced mental models about AI provenance and trust.
Technical innovations
- Empirical exploration of user mental models and prompting strategies for integrated conversational AI in a web browser context, extending beyond standalone CAI products.
- Identification of citation presence as a key factor increasing user trust and reducing motivation to fact-check in integrated CAI responses.
- Documentation of diverse, conflicting prompting strategies grounded in informal social learning rather than formal training or vendor guidance.
- Analysis of users' assumptions that conversational AI answers are generated via search engine popularity or frequency heuristics.
Datasets
- Copilot interaction transcripts — 20 participants — proprietary, not publicly available
Limitations
- Small sample size (20 university-affiliated participants, predominantly young and female) limits demographic generalizability.
- Participants already had experience with some conversational AI, potentially biasing initial mental models and usage patterns.
- No adversarial evaluation of trust or maliciously manipulated responses was conducted; real-world adversarial contexts remain unexplored.
- Study limited to a single CAI system (Microsoft Edge Copilot) and tasks; findings may differ with other platforms or task types.
- No longitudinal data; participants’ mental models and trust may evolve with extended use.
- No quantitative measures or statistical validation of thematic coding provided.
Open questions / follow-ons
- How do prompting strategies and mental models evolve over prolonged real-world use of integrated CAI agents?
- What are effective ways to design CAI interfaces and explanations that appropriately calibrate user trust without over-reliance on superficial signals like citations?
- How do diverse user populations with different expertise levels approach integrated CAI trust and usage strategies?
- Can integrated CAIs dynamically adapt explanations and citations to improve user understanding of sources and reduce misinformation risks?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this research highlights critical user trust and comprehension dynamics in integrated LLM-based conversational agents. Understanding that users often treat AI outputs like a summarized search engine result and may over-trust citations despite occasional inaccuracies suggests designing defensible user interfaces that clearly communicate provenance and uncertainty. The observed reliance on heuristic beliefs around popularity and the neutral acceptance of vendor-owned source prioritization caution against opaque AI integration without transparency. Insights into fragmented prompting strategies and trust calibration can inform security designs that mitigate automation bias and over-reliance on AI-generated information, key in contexts where bot behavior or misinformation could exploit user heuristics. Additionally, the findings encourage considering how citation design and explanation strategies impact user skepticism, a critical element in CAPTCHA and bot detection systems relying on human-AI interactive trust judgments.
Cite
@article{arxiv2605_14849,
title={ Beliefs and Misconceptions around Integrated Conversational AI },
author={ William Seymour and Adam Jenkins and Mark Cote and Jose Such },
journal={arXiv preprint arXiv:2605.14849},
year={ 2026 },
url={https://arxiv.org/abs/2605.14849}
}