Can an Old Dog Be Taught New Tricks? Taking LLMs Beyond Sentence Level Translation
Source: arXiv:2607.14040 · Published 2026-07-15 · By Alaina Brandt
TL;DR
This paper addresses the entrenched paradigm in automatic translation systems that treat translation as a sentence-by-sentence process and investigates whether Large Language Models (LLMs) can be leveraged to perform corpus-informed, whole-document translation. The authors present PAT (Pragmatic Auto-Translator), a Retrieval-Augmented Generation (RAG)-based system that incorporates user-configured specifications and relevant context from a comparable corpus of authentic long-form texts in U.S. English and Latin American Spanish. By retrieving paragraph-, section-, and document-level examples, PAT aims to produce draft translations that are reformulated to fit Spanish-language discourse organization, rhetorical style, and pragmatic norms, moving beyond literal sentence-level equivalences.
They evaluated six automatic translations of essays on generative AI across three projects, using a customized Multidimensional Quality Metrics (MQM) typology assessed by trained bilingual evaluators. Results show that limited prompts yield no meaningful reformulation, while incorporating user specifications and corpus context sometimes produces moderate to substantial reformulation, though with mixed success in terms of translation quality and effectiveness. The findings suggest that LLMs can be nudged away from sentence-level translation toward whole-document reformulation, but further refinement is necessary to improve fidelity and naturalness in the target language. The paper also contributes considerations on system design, corpus construction, and evaluation methodology tailored to pragmatic equivalence in machine translation.
Key findings
- Limited prompts instructing LLMs to translate produce essentially sentence-by-sentence translations with minimal reformulation across all tested models (Table 1 shows 20/20 limited-prompt translations were close translations).
- Reformulation prompts elicited freer translations in 13 out of 20 cases, with models differing in receptiveness: Claude Sonnet 4.6 produced 4/5 free translations but with omissions; Gemini 2.5 Pro produced 4/5 freer translations with subtle but meaningful shifts.
- Corpus-informed translation using retrieved context at paragraph, section, and document levels led to moderate to substantial reformulation and more target-language idiomatic expressions (e.g., "un sinfín de blogueros" vs "blogueros").
- Specifications-only translations produced less reformulation for broader LATAM Spanish versus more specific Mexican Spanish targets, indicating audience specificity affects reformulation.
- Excessive or poorly formatted retrieved context led to mixed effects—Project 7 showed more paragraph consolidation but less reorganization than prompt-only translation, likely due to concatenated inputs with invisible paragraph breaks.
- Interannotator agreement for error span identification (character-level F1) ranged between 49.5% and 61.4%, below target but acceptable given the subjective nature of translation evaluation.
- Evaluation highlighted the importance of combining user specifications with carefully selected, well-formatted corpus context to guide whole-document reformulation by LLMs.
- LLM model choice matters: Gemini 2.5 Pro's use of RLHF to prioritize helpfulness and instruction following yielded sustained reformulations compared to DeepSeek V4’s optimization for correctness that sometimes reverted outputs to close translation.
Threat model
The adversary modeled is the entrenched sentence-level translation bias present in LLM training data and architectures, which favors literal sentence-by-sentence translation over discourse-aware reformulations. The approach assumes access to large, non-parallel corpora and LLMs but does not consider active adversaries attempting to subvert translations or inject malicious content.
Methodology — deep read
The research explores automatic translation as a whole-document problem rather than sentence-level, focusing on English to Latin American Spanish translation in the domain of generative AI essays.
Threat Model & Assumptions: The adversary is essentially the translation paradigm embedded in LLM training and internal architecture — the sentence-by-sentence bias entrenched in datasets and prompts. The approach assumes access to an LLM and a comparable bilingual corpus. No adversarial attacks are considered.
Data: The core resource is a newly constructed comparable corpus named PAT-GAI-Longform-ESP-419-ENG-USA, consisting of approximately 200,000 words in English and 225,000 words in Spanish. Text genres include essays, reports, legal commentary, magazine articles. The corpus contains authentic (non-translated) texts to avoid source language interference and biases. The corpus covers Latin American Spanish dialects, primarily Mexican Spanish, with some Castilian Spanish and international English. Evaluation data includes six essays (~1200 words each) on generative AI, split into eight projects for model testing and annotation.
Architecture / Algorithm: PAT uses Retrieval-Augmented Generation (RAG) to combine an LLM (Gemini 2.5 Pro) with user-configured specifications and context retrieved from the corpus at multi-level document granularity (paragraph, section, document). Embeddings for retrieval use jina-embeddings-v3, a 570M parameter multilingual model based on Facebook AI’s xlm-roberta-large backbone, capable of 8k tokens context length and 1024-dimensional output. Retrieved context, along with translation specifications (subject field, language variants, text type, audience, complexity), are passed in a system prompt instructing the LLM to produce whole-document reformulated translations. The prompt avoids the word 'translate' to reduce sentence-level bias and explicitly instructs against sentence-by-sentence translation.
Training Regime: The LLM is pre-trained; no additional LLM training by the authors is conducted. Model parameters used default settings (temperature=1.0, top-p=0.95) to maintain controlled generation.
Evaluation Protocol: Translation quality is assessed using a customized Multidimensional Quality Metric (MQM) typology with additional cohesion error categories and a holistic adequacy/readability rating. Seven evaluators fluent in English and Spanish were recruited and trained on MQM; after filtering for consistency, two evaluators (U.S. English to Mexican Spanish) provided error annotations on whole, unsegmented texts via Label Studio. Eight projects were constructed, with initial training projects (0-4) and production phase projects (5-7) focusing on limited prompt, specifications-only, and corpus-informed conditions. Interannotator agreement was measured via character-level partial span F1 score, aiming for a relaxed threshold (~60%) due to inherent subjectivity.
Reproducibility: The corpus and system prompt datasheets are provided in Annex 1 with access instructions. However, the corpus is not fully public. No code repository or frozen LLM weights are provided. The embeddings model (jina-embeddings-v3) and LLM (Gemini 2.5 Pro) are commercial / proprietary.
Concrete Example: In Project 6, the source text on algorithms was retrieved with nine context passages illustrating English shorter sentence structures and Spanish complex digressions. The corpus-informed translation rendered more idiomatic, integrated and reordered Spanish text compared to prompt-only output. For example, "devoted a lot of energy" became idiomatic "han dedicado ríos de tinta" in corpus-informed but remained "han dedicado mucho tiempo y energía" in prompt-only.
In sum, the methodology rigorously combines user specifications, corpus retrieval at multiple document levels, prompt engineering, and human evaluation to probe the capacity of LLMs to move beyond sentence-level translation in a realistic, professional draft translation setting.
Technical innovations
- Demonstration that multi-level document embedding retrieval (paragraph, section, document) can be used to provide localized context to an LLM to promote pragmatic, whole-document reformulation in translation.
- Introduction of user-configured translation specifications integrated with corpus context into a system prompt to guide LLM translation away from sentence-level literalness toward stylistically and pragmatically adapted output.
- Empirical comparison of multiple LLMs showing how training objectives (e.g., RLHF versus rubric-guided RL) influence receptiveness to reformulation prompts.
- Customized MQM-based error typology including a novel cohesion error category to capture discourse-level issues in translation evaluation.
- Provision of an openly documented comparable corpus of authentic longform bilingual texts for pragmatic translation research.
Datasets
- PAT-GAI-Longform-ESP-419-ENG-USA — ~425,000 words (200k English, 225k Spanish) — comparable corpus of authentic longform U.S. English and Latin American Spanish texts collected by authors
Baselines vs proposed
- Limited prompt translation: reformulation rate = 0 of 20 translations showed meaningful reformulation vs reformulation prompt: 13 of 20 translations showed freer reformulations
- Claude Sonnet 4.6: 4/5 freer translations (but with omissions) vs Gemini 2.5 Pro: 4/5 freer translations with more subtle, meaningful reformulations
- Prompt-only translation into LATAM Spanish: minor reformulation vs prompt + corpus-informed into Mexican Spanish: moderate to substantial reformulation
- BLEU and n-gram metrics are noted as insufficient; human evaluation via customized MQM metrics show corpus-informed translations achieve higher adequacy and more idiomatic phrasing (see Table 3 for sample lexical differences)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14040.
Fig 1: Partial span agreement (F1, character-level) and category agreement as a proportion of partial span matches, measured
Limitations
- Only six essays and eight projects evaluated; limited sample size limits generalizability.
- Corpus context format issues (e.g., concatenated paragraphs with invisible breaks) reduced reformulation effectiveness in some cases (Project 7).
- Evaluations limited to English to Spanish translation; no experiments in reverse direction.
- Limited breadth of text types and domains (only generative AI essays).
- Interannotator agreement on error categories remained below 60%, reflecting inherent subjectivity and evaluation complexity.
- No adversarial robustness tests or out-of-domain generalization experiments were conducted.
- Models tested (Gemini 2.5 Pro, DeepSeek V4 Flash) and embedding models are commercially controlled, limiting reproducibility.
Open questions / follow-ons
- How can retrieval prompts and corpus context be optimized to maximize reformulation quality without introducing noise or loss of coherence?
- To what extent can similar document-level translation techniques generalize to other language pairs and diverse text domains?
- Can long-context embedding models trained specifically on document flow patterns further improve discourse-aware retrieval and LLM-guided translation?
- What objective automatic metrics can be developed to better capture pragmatic equivalence and discourse coherence beyond sentence-level BLEU scores?
Why it matters for bot defense
For practitioners in bot-defense and CAPTCHA development, this research illuminates how LLMs can be steered to handle longer-span language tasks beyond isolated sentences, which is relevant given that automated abuse and bot behavior often involve multi-turn, contextual text generation. Understanding the challenges in eliciting context-aware reformulation by LLMs could inform the design of challenge-response mechanisms that probe semantic coherence over discourse, a potential line of defense against bots relying on sentence-level translation or generation heuristics. Furthermore, insights into how retrieval-augmented approaches and user specifications influence LLM outputs may inspire more robust, context-sensitive verification tasks that are harder for bots to circumvent using naive sentence-level models.
Cite
@article{arxiv2607_14040,
title={ Can an Old Dog Be Taught New Tricks? Taking LLMs Beyond Sentence Level Translation },
author={ Alaina Brandt },
journal={arXiv preprint arXiv:2607.14040},
year={ 2026 },
url={https://arxiv.org/abs/2607.14040}
}