Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies
Source: arXiv:2607.15146 · Published 2026-07-16 · By Filippos Vlahos, Guillaume Bied, Tijl De Bie
TL;DR
This paper presents a large-scale, systematic audit of political neutrality comparing Grokipedia, an encyclopedia generated entirely by the Grok LLM, against Wikipedia. Motivated by debates claiming Wikipedia exhibits left-leaning bias, the authors investigate whether an LLM-written encyclopedia can offer a more neutral perspective or simply reflects a different ideology. The study analyzes 1,394 matched article pairs about government members using expert-coded ideological scores across nine dimensions and solicits neutrality ratings from a diverse panel of four state-of-the-art LLM judges (Grok, Claude, Mistral, DeepSeek). The results show that neither source is fully neutral. Grokipedia is rated as less neutral overall than Wikipedia, with a particularly favorable bias towards economically right-wing politicians and less favorable treatment of socially liberal politicians. Conversely, Wikipedia favors socially liberal politicians. Importantly, even Grok—the LLM behind Grokipedia—rates Grokipedia as less neutral than Wikipedia, indicating that Grokipedia’s ideological patterns are inherent to its content rather than merely rater bias. Additionally, the study finds substantial variability in neutrality judgements between LLM judges, highlighting the importance of using multiple evaluators to detect nuanced ideological bias. This work offers a novel, scalable methodology grounding bias audits in expert-coded ideological dimensions and diverse LLM raters, yielding insights relevant for the growing ecosystem of AI-generated knowledge bases.
Key findings
- All four LLM judges rate Grokipedia as less neutral than Wikipedia across 1,394 politician article pairs.
- 39% of Grokipedia articles are rated biased vs 29.7% for Wikipedia, with Grokipedia articles more frequently rated as favorably biased.
- Grokipedia favors economically right-wing politicians (β=+0.221; p<0.05) and politicians supporting political pluralism (β=+0.119), but disfavors socially liberal politicians (e.g., LGBT rights β=-0.095, women's labor equality β=-0.050).
- Wikipedia favors socially liberal politicians, e.g., LGBT rights (β=+0.077, p<0.05) and immigration openness (β=+0.036).
- Economic right-wing leaning is the strongest ideological predictor of bias differences between the two encyclopedias (β=-0.226 for Wikipedia-Grokipedia neutrality gap).
- Inter-annotator agreement among LLM judges on exact neutrality ratings is low (~24%), but direction agreement is high (~88%), underscoring the benefit of multiple raters.
- Grok, the model behind Grokipedia, assigns higher bias scores to Grokipedia content versus Wikipedia (mean absolute bias difference Δ=0.041), confirming intrinsic bias in Grokipedia.
- The ideological characteristics explain a substantially larger share of variance in neutrality ratings for Grokipedia (R²=0.220) compared to Wikipedia (R²=0.059), indicating stronger ideology-driven bias in Grokipedia content.
Threat model
Not applicable; this is an audit and measurement study rather than a security-focused paper. The adversarial aspect is implicitly the risk of ideological bias embedded by encyclopedia creators or model developers, but no explicit attacker capabilities are modeled.
Methodology — deep read
The authors formulated a rigorous methodology to audit political bias in Grokipedia and Wikipedia at scale. They compiled a dataset of articles about members of government from 2016–2023, starting with the WhoGov dataset listing 9,752 politicians in 177 countries. Each politician was mapped to nine expert-coded political ideology dimensions from the V-Party dataset, which provides validated party positions on key cultural, social, and economic issues (e.g., cultural inclusivity, immigration, LGBT rights, economic right-wing). This approach avoids circularity of scoring ideology with LLMs themselves. Matching article pairs on Grokipedia (version 0.2) and Wikipedia were collected, yielding 1,394 article pairs after filtering for coverage, length (>300 words), and confirmation of subject relevance.
Neutrality judgments were obtained from four diverse LLMs (Grok-4, Claude-Opus-4.6, Mistral-Medium-3.5, DeepSeek-V4-Pro) selected to span ideological and geographical variation and include the generative model behind Grokipedia (Grok). The LLM judges were prompted with a formal neutrality definition inspired by Wikipedia’s Neutral Point of View policy, emphasizing fair representation of viewpoints, avoidance of persuasive language, judgmental terms, and fringe views presented as facts. Each article was rated on a 5-point Likert scale from “strongly biased against” (-2) through neutral (0) to “strongly biased in favor” (+2), with an accompanying explanation for grounding. The prompt’s sensitivity was tested by re-rating a subset without the neutrality definition, confirming robustness of ratings.
Statistical analysis involved two linear regression specifications. First, the neutrality differential between Wikipedia and Grokipedia ratings for the same politician-judge pair was regressed on the politician’s ideological vector to identify how ideological dimensions predict relative favorability. Second, separate models regressed individual neutrality ratings within each encyclopedia on ideological scores and LLM judge indicator variables to characterize encyclopedia-specific biases and judge tendencies. Standard errors were clustered at the politician level to account for dependence. Multicollinearity was checked via Variance Inflation Factors, all below 5, confirming independent ideological dimensions.
Inter-annotator agreement between LLM judges was measured using Cohen’s kappa and Krippendorff’s alpha to quantify consistency in bias direction and strictness. Additionally, descriptive statistics and bias magnitude comparisons were reported.
No human annotations were directly involved in neutrality rating; the reliance on LLM judges reflects a tradeoff favoring scalability and consistency over possible human variability, with the authors acknowledging alignment verification as future work. The approach was applied end-to-end to 1,394 politician article pairs, leveraging the ideological vectors from the V-Party dataset to ground neutral assessment in political theory rather than model-generated cues.
Technical innovations
- Introduction of a large-scale, multi-dimensional political bias audit framework for encyclopedias grounded in expert-coded ideology dimensions (V-Party), avoiding circular LLM-based ideology scoring.
- Use of a diverse panel of four state-of-the-art LLM judges spanning ideological and geographical spectra—including the generative model behind Grokipedia—to assess neutrality with explicit grounding prompts.
- Application of neutrality differential regression comparing matched article pairs to isolate ideological biases inherent to encyclopedias rather than raters.
- Demonstration of how LLM judges differ systematically in bias strictness but tend to agree on bias direction, motivating multi-judge ensembles for more balanced assessments.
Datasets
- WhoGov — 9,752 members of government from 177 countries (2016-2023) — public
- V-Party — expert-coded ideology scores for political parties (1970-2019) — public
- Article pairs — 1,394 matched Grokipedia and Wikipedia articles on politicians with >300 words, verified for content relevance — curated
Baselines vs proposed
- Wikipedia: mean absolute bias magnitude = 0.707 (Claude), 0.383 (Grok), 0.192 (Mistral), 0.127 (DeepSeek) vs Grokipedia: 0.946 (Claude), 0.424 (Grok), 0.284 (Mistral), 0.203 (DeepSeek)
- Neutrality rate (Wikipedia vs Grokipedia, Claude judge): 34.7% vs 15.2%
- Neutrality rate (Wikipedia vs Grokipedia, average over 4 judges): 70.3% vs 61.0%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15146.
Fig 1: Methodology overview: members of government (WhoGov, 20162023) are
Fig 2 (page 2).
Fig 3 (page 2).
Fig 5: Heatmap of country-origin of the members of government in the final dataset.
Limitations
- Neutrality is operationalized via a specific Wikipedia-inspired definition, which may not capture all facets or conceptions of neutrality.
- LLM judges are known to carry inherent ideological biases and may apply subjective or inconsistent standards; human alignment of ratings remains unverified here.
- The analysis uses articles as they appeared at a single snapshot in time, ignoring temporal content changes or updates.
- Mapping individual politicians to party-level ideology scores assumes ideological homogeneity which may not hold perfectly for all individuals.
- The study focuses exclusively on politicians and government members, limiting generalizability to other article topics or domains.
- Evaluations do not consider adversarial attempts to manipulate content or raters, nor the impact of improved Grokipedia versions beyond v0.2.
Open questions / follow-ons
- How do human expert judgments of bias compare to those from LLM judges in this framework?
- Can prompt engineering or fine-tuning reduce ideological bias in generative LLM encyclopedias like Grokipedia?
- Does the ideological bias of Grokipedia reflect training data biases, prompting methods, or inherent model tendencies?
- How stable are neutrality and bias ratings under temporal changes in encyclopedia content or updates in LLM judge models?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this paper highlights the complexities in automatically auditing ideological bias in AI-generated content using LLMs themselves as evaluators. While not directly about bot detection, the multi-LLM judge panel approach and methodology for disentangling source-specific biases is instructive for designing automated integrity evaluations of online content or knowledge bases. Practitioners aiming to ensure platform neutrality or detect propaganda risks in user-generated or AI-generated encyclopedia-like data may adapt similar expert-grounded dimensions and multi-model assessment strategies. It also underscores the need to carefully select and cross-validate AI judges to avoid confirmation bias. Finally, insights about systematic ideological tendencies encoded by LLMs that underlie widely cited knowledge sources are relevant background for understanding potential influence vectors exploited by malicious bots or state actors spreading politically slanted content through trusted channels.
Cite
@article{arxiv2607_15146,
title={ Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies },
author={ Filippos Vlahos and Guillaume Bied and Tijl De Bie },
journal={arXiv preprint arXiv:2607.15146},
year={ 2026 },
url={https://arxiv.org/abs/2607.15146}
}