ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Source: arXiv:2607.02509 · Published 2026-07-02 · By Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen et al.

TL;DR

This paper addresses a key challenge in long-context large language model (LLM) use: although models can ingest very long contexts (up to 128K tokens), they often fail to utilize relevant evidence contained within those contexts effectively during generation. The authors propose RECONTEXT, a training-free inference-time method that harnesses internal model attention to identify and recursively replay question-relevant evidence spans extracted from the original long context before final answer generation. This recursive evidence replay dynamically emphasizes relevant information without pruning or altering the model's backbone, improving long-context reasoning. Theoretical analysis interprets the process as associative memory trace reactivation, demonstrating monotonic improvement in similarity between hidden states and answer embeddings. Experiments across eight diverse 128K-token datasets and three LLM backbones (Qwen3-4B, Qwen3-8B, Llama3-8B) show RECONTEXT consistently outperforms strong baselines, boosting average accuracy across tasks by 24.6% relative over vanilla full-context prompting. Ablations explore recursion rounds and evidence token budgets, and qualitative examples confirm more grounded, evidence-based answers. RECONTEXT offers an elegant, lightweight inference-time scaffold for better mounting relevant knowledge in very long contexts.

Key findings

• RECONTEXT improves mean accuracy across eight long-context 128K-token datasets and three backbone LLMs from 0.24 to 0.30, a 24.6% relative gain over vanilla prompting.
• It achieves the best average method rank on every backbone evaluated: 1.00 on Qwen3-4B, 1.46 on Qwen3-8B, and 1.29 on Llama3-8B.
• On Qwen3-4B, RECONTEXT raises Natural Questions accuracy from 0.02 (best baseline) to 0.08.
• Recursive rounds of evidence selection improve accuracy, with 2-4 rounds optimal depending on dataset/task.
• Selecting candidate evidence spans solely from the original long context outperforms selecting from the full replay prompt, boosting macro-average accuracy from 0.19 to 0.23.
• RECONTEXT maintains full original context during generation, emphasizing rather than pruning input, thus preserving access to all information.
• Theoretical monotonic improvement proof shows each evidence replay round increases cosine similarity between hidden generation state and answer embedding.
• RECONTEXT adds modest runtime overhead compared to vanilla prompting but is substantially faster than methods modifying decoding or backbone forward passes.

Methodology — deep read

1. Threat Model & Assumptions: The adversary setting is implicit; the work focuses on improving inference utilization of internal evidence rather than adversarial robustness. The model is assumed to be a frozen LLM with internal attention weights accessible during inference. The method assumes the question and full context are provided, and that internal model signals correlate with relevance.

2. Data: Evaluation uses eight public or previously established long-context datasets, all adapted to 128K-token contexts: Natural Questions (NQ), TriviaQA, HotpotQA, PopQA, NarrativeQA, InfBench QA, InfBench MC, and CLIPPER. These datasets contain factual QA, multi-hop questions, narrative understanding, and claim verification. Standard train/dev/test splits and accuracy or F1 metrics apply. Data preprocessing applies the HELMET method to extend context lengths.

3. Algorithm & Architecture: RECONTEXT operates as a training-free inference wrapper around the backbone LLM M. It uses the last w=8 tokens of the prompt suffix (the query cues) to extract internal relevance scores for all tokens in the original context. These scores aggregate attention weights over selected heads and layers, normalized with exponential decay to yield token relevance.

The top-K tokens (e.g., K=8 to 32) with highest scores form candidate evidence seeds. These seeds are mapped to sentence-level spans covering those tokens to form an evidence pool. The evidence pool text is then prepended ("replayed") alongside the full original context and question to produce the next prompt, conditioning generation indirectly toward this emphasized evidence.

This evidence selection and replay process is iterated R times (typically 2-4 rounds). Each round conditions attention relevance on both the original context and the cumulative evidence pool built so far, thus recursively refining and expanding the evidence collection before the final generation call.

4. Training & Hyperparameters: No training or fine-tuning is done; the backbone models remain frozen. Prompting and decoding hyperparameters are held constant across comparisons. Recursive rounds R and candidate token budgets K are tuned in ablations. ReContext adds a lightweight evidence-selection pass with negligible memory overhead.

5. Evaluation Protocol: RECONTEXT is compared against multiple strong baselines including vanilla full-context generation, AttnSharp (attention sharpening), DySCO (dynamic scaling of decoding attention), A-MEM (agentic external memory), and DAC (dynamic attention compression). Metrics include accuracy and token-level F1 as per dataset standards. Average method rank across all datasets is computed to aggregate performance. Ablations vary recursion rounds and K to study tradeoffs. Theoretical validity is supported by a monotonic improvement proof (Theorem 1). Qualitative case studies highlight sample effects.

6. Reproducibility: Code and models are publicly released on the authors' GitHub, though the datasets used are large benchmarks publicly available elsewhere. The method requires access to model internal attention values, so use with closed APIs that hide these signals is limited.

Example workflow: Given context C and query q, start with empty evidence pool E(0). Compute internal relevance scores r(1) over C for q, select top-K tokens, materialize to spans bS(1), update evidence pool E(1) = E(0) ∪ ΔE(1). Construct prompt x(1) = [C; ϕ(E(1)); q]. Repeat up to R rounds, each time recomputing scores conditioned on new scaffold. After R-th round, generate answer from prompt x(R). This recursive expansion turns sparse, internally identified relevant tokens into explicit textual context that steers answer generation while preserving access to the full original input.

Technical innovations

• A novel training-free inference method (RECONTEXT) that recursively selects and replays question-conditioned evidence spans from full long contexts using internal LLM attention signals.
• A process that separates evidence organization from answer generation by maintaining an evolving evidence pool prepended during inference without pruning or retraining.
• Theoretical framing of long-context prompting as associative memory retrieval with a monotonic improvement guarantee via recursive cue-trace rebinding and trace reactivation.
• Demonstration that selecting evidence only from the original context for replay outperforms selection from expanded replay prompts, emphasizing the importance of grounding.

Datasets

• Natural Questions (NQ) — 128K context version — constructed by HELMET
• TriviaQA — 128K context version — constructed by HELMET
• HotpotQA — 128K context version — constructed by HELMET
• PopQA — 128K context version — constructed by HELMET
• NarrativeQA — 128K context version — constructed by HELMET
• InfBench QA — 128K context version — constructed by HELMET
• InfBench MC — 128K context version — constructed by HELMET
• CLIPPER — 128K context book contexts — public claim verification benchmark

Baselines vs proposed

• Vanilla full-context prompting: average accuracy = 0.24 vs RECONTEXT: 0.30 (+24.6% relative gain)
• AttnSharp (attention sharpening): lower average rank than RECONTEXT across all backbones
• DySCO (dynamic decoding attention scaling): lower average rank and accuracy than RECONTEXT
• A-MEM (external memory retrieval): lower average rank and accuracy than RECONTEXT
• DAC (dynamic attention compression): lower accuracy and average rank than RECONTEXT
• On Qwen3-4B/Natural Questions, Vanilla accuracy 0.02 vs RECONTEXT 0.08
• On Qwen3-8B/HotpotQA, RECONTEXT accuracy 0.68 vs best baseline 0.65
• On Llama3-8B/NarrativeQA, RECONTEXT accuracy 0.22 vs DAC 0.16 and Vanilla 0.21

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02509.

Fig 1: Top 0.1% of context tokens already ac-

Fig 2: Overview of RECONTEXT. RECONTEXT identifies question-relevant evidence from a long context using

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Requires access to internal LLM attention weights or relevance signals, limiting applicability with closed-source or API-only models without introspection.
• Adds an extra inference evidence-selection and replay stage, resulting in higher latency than direct full-context decoding, though still faster than methods modifying core attention.
• Does not perform adversarial analysis or robustness testing against malicious inputs or manipulative evidence.
• Evaluations are limited to 128K-token contexts and specific knowledge/ref QA benchmarks; generalization beyond tested tasks is unproven.
• Recursive rounds R and evidence token budget K are hyperparameters that need tuning per task.
• Theoretical results rely on simplifying assumptions about embeddings and attention behaving like associative memory, which may not fully hold in practice.

Open questions / follow-ons

• How well does RECONTEXT scale or adapt to significantly longer contexts beyond 128K tokens, e.g., million-token regimes?
• Could learned adaptation or fine-tuning during evidence selection further improve recursive relevance rather than relying solely on frozen internal signals?
• How does RECONTEXT perform with multimodal long contexts or non-textual inputs when relevance signals are less clear?
• What robustness guarantees or failure modes arise if input contains misleading or adversarial evidence spans?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, RECONTEXT's approach presents a novel inference-time mechanism to more reliably harness relevant evidence embedded in very long contexts without pruning or retraining models. This can inform design of LLM-based verification systems requiring robust use of extensive context (e.g., large multi-step interactions or historical logs). By recursively replaying internally identified relevant spans, one can potentially reduce hallucinations and improve answer grounding under heavy context loads, which is critical for trustworthy automated challenge-responses. However, practical application requires model internals access, so applicability depends on deployment environment. The associative-memory interpretation may inspire new dynamic context management strategies for bot-defense LLM pipelines, encouraging separation of evidence retrieval and final answer generation. RECONTEXT also highlights a complementary direction to retrieval augmentation methods by focusing on harnessing existing context signals rather than external retrieval or compression.

Cite

@article{arxiv2607_02509,
  title={ ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning },
  author={ Yanjun Zhao and Ruizhong Qiu and Tianxin Wei and Yuanchen Bei and Zhining Liu and Lingjie Chen and Ismini Lourentzou and Hanghang Tong and Jingrui He },
  journal={arXiv preprint arXiv:2607.02509},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02509}
}

Read the full paper

• arXiv:2607.02509 (abstract)
• arXiv:2607.02509 (PDF)