How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

Source: arXiv:2606.27275 · Published 2026-06-25 · By Maria Levchenko

TL;DR

This paper investigates the challenges large language models (LLMs) face when processing historical texts, focusing on Italian from the 17th century. Prior work often treats "historical difficulty" as a single problem conflating orthographic variation, linguistic distance, and pretraining exposure. Levchenko introduces a diagnostic framework that separates this difficulty into four dimensions: tokenization cost ("tokenization tax"), predictive uncertainty measured by perplexity ("comprehension tax"), semantic robustness via embedding similarity, and contextual adaptability through temporal prompting. Using a novel 17th-century Italian corpus digitized from original page images, a canonical 19th-century Italian text (Manzoni's I Promessi Sposi) as a high-exposure control, and 18th-century Russian civil print books as an orthographic contrast, the study systematically quantifies these effects.

Key findings include a dissociation between tokenization cost and comprehension difficulty—both 17th-century Italian and 18th-century Russian texts incur similar 25-30% token inflation, but Italian historical texts have 2.4x higher perplexity indicating greater predictive uncertainty. Despite this, embedding similarity remains robust (>0.85) across datasets, meaning models capture semantic meaning even when generation is challenging. Crucially, a minimal temporal context prompt indicating the text's date reduces surprisal by approximately 58%, demonstrating a simple, model-agnostic mitigation strategy. The work highlights that historical orthographic variation imposes a mechanical encoding tax distinct from linguistic comprehension difficulty, and that temporal priming significantly reduces comprehension costs while preserving semantic representation. This nuanced understanding informs deploying LLMs in digital libraries to safely use semantic retrieval on historical texts while adapting generative applications.

Key findings

• 17th-century Italian texts cause a +27.5% tokenization inflation compared to modernized baseline, mainly driven by the long s (ſ) character.
• 18th-century Russian texts incur a similar tokenization inflation of +25-36% due to obsolete characters causing byte-level fallback.
• Mean perplexity ratio (historical/modern) for 17th-century Italian is 2.42×, with academic prose reaching 3.22×, indicating high predictive difficulty.
• 18th-century Russian texts have a much lower perplexity increase at 1.25× despite similar token inflation.
• Manzoni’s 19th-century canonical Italian shows negligible tokenization inflation and only a 1.70× perplexity increase, acting as a high-exposure control.
• Minimal temporal context prompts prepended to text reduce surprisal by approximately 58% for 17th-century Italian and 17% for 18th-century Russian.
• Embedding similarity between historical and modern sentence pairs remains high across datasets (≥0.85), with Qwen 2.5 achieving 0.938 cosine similarity on 17th-century Italian, indicating preserved semantic representations.
• Tokenization inflation correlates poorly with perplexity (difficulty) and embedding similarity, underscoring the dissociation between encoding cost and comprehension.

Threat model

n/a — The paper does not address adversarial threat scenarios but treats historical language variation as a domain mismatch problem causing distributional shifts in tokenization and predictive uncertainty in pretrained language models.

Methodology — deep read

The study evaluates historical language difficulty with a multi-dimensional framework distinguishing tokenization cost, predictive difficulty (measured via perplexity), semantic robustness (measured via embedding similarity), and context sensitivity (measured by incorporating temporal prompts).

1. Threat Model & Assumptions: The adversary is not explicitly defined as a security threat model, but rather the challenge is a linguistic domain shift—modern LLMs pretrained largely on contemporary data face unknown orthographic, lexical, and syntactic distributions in historical texts. The adversary effect is intrinsic: the historical texts differ temporally and orthographically from training corpora, causing token and distributional mismatches. There is no known adversarial manipulation or direct exposure of these specific scanned editions in the pretrained data.

2. Data: Three datasets were used covering three centuries: (i) a newly curated corpus of five 17th-century Italian texts (1610-1689) digitized from original page images via custom OCR, yielding 732 aligned sentence pairs (~15,000 tokens), with manual semantic normalizations verified by experts; (ii) Manzoni's 19th-century canonical Italian dataset (Quarantana 1840–42 edition) of ~400 sentences paired with modernized versions, acting as a high-exposure control; (iii) 18th-century Russian civil print books dataset comprising 374 aligned sentence pairs of pre-reform orthography, preserved special characters, and diverse genres. These datasets were chosen to isolate orthographic vs linguistic difficulty.

3. Architecture / Algorithm: The primary language model used for evaluation was Qwen 2.5 1.5B—a multilingual transformer model with a unigram or BPE tokenizer. Tokenization was analyzed with the OpenAI 200k-base tokenizer and compared to Qwen’s tokenizer. Embedding similarity was calculated using multiple models: OpenAI text-embedding-3-large, Google LaBSE, and the Qwen 2.5 hidden states. The metrics computed were:

   • Token inflation = relative increase in token count of historical vs modernized text under fixed tokenizer.
   • Perplexity = exponentiated negative log likelihood of next-token predictions normalized by length, computed sentence-wise and aggregated.
   • Embedding similarity = cosine similarity of sentence embeddings from historical/modern aligned pairs.
   • Context sensitivity = change in perplexity after prepending minimal temporal prompt strings.

4. Training Regime: Model training was not conducted; off-the-shelf pretrained checkpoints were used to isolate evaluation effects without fine-tuning or adaptation.

5. Evaluation Protocol: Perplexity was computed using paired historical-modern sentence comparisons, reporting mean historical to modern perplexity ratios for robustness. Token inflation was compared across tokenizers and normalized orthographic variants. Ablations included partial orthographic normalization to isolate the effect of long s vs vocabulary/syntax. Contextual prompting effects were measured by adding minimal temporal label prompts and re-computing perplexity. Embedding similarity was averaged across sentence pairs and correlated to perplexity, token inflation, and sentence length. Cross-dataset comparisons were emphasized to separate encoding vs comprehension issues.

6. Reproducibility: The novel 17th-century Italian corpus is derived from publicly available FICLIT Digital Library; code specifics are not detailed, but tokenizer vocabularies and Qwen model checkpoints are publicly accessible. The Russian and Manzoni datasets have established provenance. Manual verification steps for semantic normalization are described but not fully automated, challenging exact reproducibility. Overall, the methodology is transparent but implementation details on software and seeds are not exhaustively provided.

Example: For the 17th-century academic Miscellanea subset (1687), the original text had +28.2% token inflation and a perplexity ratio of 3.22× versus its modern baseline. A conservative normalization replacing only the long s (ſ → s) increased perplexity ratio to 3.63×, showing syntactic and lexical factors dominate. Adding a temporal prompt reduced perplexity by 43%, demonstrating domain mismatch mitigated surprisal. Embedding similarity remained high (~0.85), confirming semantic preservation despite high predictive uncertainty.

Technical innovations

• Proposed a decompositional framework quantifying historical language difficulty into tokenization cost, predictive uncertainty, semantic robustness, and context sensitivity.
• Curated a new 17th-century Italian corpus digitized via OCR directly from page images with manual semantic normalization for robust historical language evaluation.
• Demonstrated a clear dissociation between tokenization inflation and perplexity, showing that encoding cost is a poor proxy for comprehension difficulty.
• Introduced a minimal, model-agnostic temporal prompting strategy to reduce historical text surprisal by approximately 58% without fine-tuning.

Datasets

• 17th-Century Italian Corpus — 732 aligned sentence pairs (~15,000 tokens) — Derived from FICLIT Digital Library, University of Bologna
• Manzoni Quarantana Edition (1840–42) — ~400 sentences — Public canonical Italian literature digitized by LeggoManzoni project
• 18th-Century Russian Civil Print Books — 374 aligned sentence pairs — National Library of Russia historical OCR benchmark

Baselines vs proposed

• Manzoni Quarantana: perplexity ratio = 1.70× vs 17th-century Italian corpus: 2.42×
• Russian civil print books: perplexity ratio = 1.25× vs 17th-century Italian corpus: 2.42×
• Russian civil print books: token inflation = 25-36% vs 17th-century Italian: ~27.5%, yet perplexity differs widely
• No prompt context: 17th-century Italian perplexity ratio = 2.42× vs with temporal prompt: reduction by 58%
• No prompt context: Russian perplexity ratio = 1.25× vs with temporal prompt: reduction by 17%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27275.

Fig 1: Dissociation of Encoding Cost and Predictive Difficulty. Comparing token inflation (x-axis) vs.

Limitations

• Exposure confound cannot be fully ruled out as exact pretrained corpus contents are proprietary, though minimized by sourcing rare historical texts.
• Evaluation relies on a single general-purpose LLM (Qwen 2.5) for perplexity; results might vary with different model families.
• OCR errors in 17th-century Italian corpus could influence token and perplexity estimates despite manual verification.
• Semantic normalization uses a mix of automated GPT-5-mini and manual checks, potentially introducing normalization biases.
• Does not evaluate active adversarial attacks or deliberate obfuscation scenarios—focus is on linguistic domain shift only.
• Reproducibility limited by lack of fully public code and detailed training/evaluation pipelines, especially for normalization steps.

Open questions / follow-ons

• Can fine-tuning or adapter methods further reduce the comprehension tax beyond temporal prompting?
• How do other LLM architectures and tokenizers respond to historical orthographic and syntactic shifts?
• What is the interplay between OCR error patterns and LLM surprisal in archival digitization workflows?
• Could multi-modal contextual prompts incorporating metadata about genre, author, or domain improve adaptation?

Why it matters for bot defense

This work provides a nuanced understanding relevant to bot-defense and CAPTCHA practitioners interested in natural language understanding over temporally varied corpora. The distinction between tokenization cost and comprehension difficulty highlights that computational burden alone does not predict interpretative challenges for LLMs. For CAPTCHA systems using language models to detect automated access on historical or obfuscated text inputs, relying solely on token fragmentation metrics is insufficient; perplexity and semantic stability metrics provide deeper insight. Additionally, the demonstrated temporal prompt mitigation indicates that explicitly supplying contextual metadata about text provenance can enhance model reliability and reduce false positive detection of difficulty. Practitioners designing text-based bot detection in archiving or digital libraries would benefit from integrating such context-aware prompting to better differentiate genuine language complexity from tokenization artifacts, ensuring robust evaluation in legacy-text scenarios.

Cite

@article{arxiv2606_27275,
  title={ How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation },
  author={ Maria Levchenko },
  journal={arXiv preprint arXiv:2606.27275},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27275}
}

Read the full paper

• arXiv:2606.27275 (abstract)
• arXiv:2606.27275 (PDF)