LLMs for Qualitative and Mixed-Methods Social Network Analysis
Source: arXiv:2607.14045 · Published 2026-07-15 · By Moses Boudourides
TL;DR
This manuscript investigates the role of Large Language Models (LLMs) in enhancing qualitative and mixed-methods social network analysis (SNA), emphasizing augmentation rather than replacement of human researchers. It articulates how LLMs’ capabilities in natural language understanding enable them to assist tasks like data collection, coding, memoing, theory-building, and abductive reasoning, thereby scaling qualitative inquiry to larger datasets and complex social phenomena. The paper highlights the foundational importance of meaning in social ties, narratives, and relational identities within qualitative SNA, which often challenges purely quantitative approaches.
Significantly, the manuscript provides practical research designs integrating LLMs with qualitative methods, such as sequential qual→LLM→qual workflows and parallel human-LLM analyses that employ intercoder reliability and disagreement analysis to preserve interpretive rigor. It also critically discusses key limitations of LLMs, including bias, hallucination, semantic homogenization, and the irreplaceable role of human reflexivity and interpretive authority. Finally, the author offers recommendations on transparency, audit trails, ethical use, and software workflows to responsibly incorporate LLMs in qualitative and mixed-methods SNA research.
Key findings
- LLMs can classify relational valence (e.g., power, friendship) from text with improving reliability, but accuracy varies significantly by model generation and task context (Argyle et al., 2023).
- Intercoder reliability protocols require human-machine agreement of at least Cohen's κ = 0.70 and <20% disagreement on validation samples to proceed with LLM-assisted coding.
- LLM-assisted analysis risks semantic homogenization by smoothing over local, minority, or context-specific tie meanings into dominant categories, necessitating manual checks to preserve emic diversity.
- LLMs excel at entity extraction and surface relational patterns at scale but cannot replace human researchers’ theoretical sensitivity and interpretive judgment for relational meaning.
- LLMs enable scaling qualitative SNA beyond small bounded networks by automating laborious coding and memoing tasks, allowing researchers to analyze thousands to millions of interactions.
- Sequential mixed-methods designs (qual→LLM→qual) combine initial qualitative insight with large-scale LLM analysis and confirmatory qualitative follow-up for richer interpretations.
- Parallel human-LLM analyses with systematic disagreement analysis reveal nuances and coding ambiguities that improve overall analytic validity.
- Effective audit trails in LLM-augmented research must record exact model versions, prompts, hyperparameters, timestamps, and archive raw outputs to guard reproducibility.
Threat model
The paper does not define a traditional security adversary but considers methodological threats introduced by LLM integration, including biased or hallucinated outputs, semantic flattening, and loss of interpretive validity. The human researcher is assumed to act as a gatekeeper who validates and corrects LLM results to protect analytic quality.
Methodology — deep read
The paper’s methodology is primarily conceptual and methodological rather than empirical, focusing on how LLMs can be integrated into qualitative and mixed-methods social network analysis (SNA).
Threat model & assumptions: The adversary here is not a malicious agent but methodological risks such as bias, hallucination, and semantic flattening introduced by LLMs. The human researcher is assumed to retain ultimate interpretive authority and ethical reflexivity. LLM outputs are treated as fallible tools needing validation.
Data provenance and scale: The text data analyzed in qualitative SNA is typically large corpora of interviews, ethnographic field notes, organizational communications, or online social texts that contain relational content. Datasets can range from deep small-N network case studies to massive multi-thousand message corpora.
Architecture/algorithm: While the paper does not introduce a new LLM architecture, it describes LLMs as pretrained language models capable of entity extraction, relational classification, coding, memoing, and generating abductive reasoning prompts. Their inputs are textual relational data, and outputs are interpretive annotations or relational categories supporting iterative analysis.
Training regime: The paper notes that existing LLMs are trained on massive corpora (internet-scale data) leading to representational bias and hallucination; it underscores the researcher’s role in correction. No retraining or fine-tuning is detailed.
Evaluation protocol: The paper recommends human-machine intercoder reliability testing using metrics such as Cohen's kappa or Krippendorff’s alpha on validation subsets, with set thresholds (κ > 0.70) to ensure sufficient agreement. Disagreement analysis is emphasized for analytic insight rather than error reduction alone. Sequential and parallel mixed-methods designs integrate qualitative validation and triangulation to maintain rigor.
Reproducibility: The paper stresses the challenges posed by nondeterministic LLM outputs, API changes, and model deprecation. It recommends precise logging of model versions, prompt text, hyperparameters, and archiving raw and corrected outputs to enable auditing.
One concrete example workflow described is a sequential qual→LLM→qual design: starting with qualitative interviews to extract themes, using LLMs to code large corpora guided by those themes, then returning to qualitative analysis to interpret key cases highlighted by the LLM outputs. Human researchers compare their manual codes with LLM outputs, iteratively refining prompts and coding schemes based on intercoder reliability results.
Technical innovations
- Framing LLMs as relational text analyzers that augment qualitative SNA by extracting nuanced tie types, relational valence, and emergent narrative patterns from large text corpora.
- Proposing a methodology combining human interpretive authority with LLMs’ scalable coding and memoing capabilities within sequential and parallel mixed-methods designs.
- Introducing human-machine intercoder reliability protocols with analytic disagreement as an opportunity for model and scheme refinement rather than mere error correction.
- Highlighting the novel challenge of semantic homogenization inherent to LLM probabilistic generation and providing concrete strategies to mitigate it in qualitative SNA.
Limitations
- LLMs currently reproduce and potentially amplify social biases present in training data, which must be critically managed.
- Semantic homogenization risks flattening culturally specific and minority relational meanings into dominant categories, reducing qualitative richness.
- Hallucination leads to plausible but factually incorrect or nonsensical outputs that require human vetting.
- Nondeterministic output and API changes challenge reproducibility and auditability of research produced with LLMs.
- LLMs cannot replace the researcher’s interpretive judgment, reflexivity, and ethical accountability foundational to qualitative inquiry.
- No empirical benchmarking or adversarial testing of LLM outputs against human-coded qualitative SNA is presented.
Open questions / follow-ons
- How do different LLM architectures and model sizes vary in their reliability for relational valence classification and tie meaning extraction?
- What are systematic evaluation benchmarks comparing LLM-assisted coding versus expert human coding in qualitative SNA?
- How can semantic homogenization be quantitatively measured and mitigated across model generations?
- Can methods for explainability or uncertainty quantification in LLMs improve interpretive confidence in qualitative network analysis?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners interested in social network analysis to detect fraud, influence, or coordinated behavior, this paper highlights that LLMs can support qualitative and mixed-methods approaches to understanding relational context at scale. Rather than automating full interpretation, LLMs can accelerate the coding of textual communication and identify anomalous relational patterns that merit human scrutiny. Incorporating LLMs enables scaling of nuanced social network insights beyond simple quantitative graphs, especially in large conversational or behavioral datasets.
Practitioners should heed the paper's caution on LLM biases, hallucinations, and semantic blending which could obscure subtle but important patterns critical to bot detection. Maintaining researcher oversight, transparent versioning, and disagreement analysis is vital to ensure robust, defensible network conclusions when integrating LLM outputs. This work encourages a complementary role for LLMs as analytic collaborators in complex social network tasks rather than black-box solvers.
Cite
@article{arxiv2607_14045,
title={ LLMs for Qualitative and Mixed-Methods Social Network Analysis },
author={ Moses Boudourides },
journal={arXiv preprint arXiv:2607.14045},
year={ 2026 },
url={https://arxiv.org/abs/2607.14045}
}