Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering

Source: arXiv:2607.21571 · Published 2026-07-23 · By Zikui Cai, Kaushal Janga, Tan Dat Dao, Seungjae Lee, Shivin Dass, Mingyo Seo et al.

TL;DR

This paper addresses a critical gap in Embodied Question Answering (EQA) research: the common episodic evaluation paradigm treats each question as an isolated episode, resetting internal memory between tasks. This is misaligned with real-world scenarios where robots operate continuously and must accumulate and reuse knowledge from prior interactions to answer sequential questions more efficiently.

The authors propose Sequential-EQA, a new evaluation protocol that preserves agent memory across multiple questions in the same scene, exposing architectural bottlenecks that episodic benchmarks hide. Through rigorous experiments on four representative EQA agents with diverse memory mechanisms—2D occupancy maps, episodic semantic libraries, structured 3D spatial memory, and implicit latent states—they find that naive memory persistence rarely improves accuracy and sometimes degrades it due to semantic interference or temporal mismatch. Only spatially grounded 3D metric memory architectures, exemplified by 3D-Mem, break this tradeoff by enabling coherent accumulation of visual-semantic evidence across queries.

Extensive simulation studies show 3D-Mem achieves a +33.3% accuracy gain and a +53.3% navigation cost reduction under sequential evaluation, uniquely combining higher answer quality with lower exploration effort. These insights generalize to physical robots tested across indoor and outdoor environments, confirming real-world relevance. Overall, the study rigorously diagnoses limitations of existing EQA memory architectures for continuous operation and establishes spatially structured 3D memory as necessary for long-horizon knowledge accumulation in embodied agents.

Key findings

• Memory persistence alone does not guarantee knowledge accumulation or answer quality improvements; agents like ExploreEQA and UniNavid observe <3% accuracy gain.
• 3D-Mem, with spatially grounded 3D visual-semantic memory, achieves +33.3% memory advantage (MA) in accuracy and +53.3% step advantage (SA) in navigation cost under Sequential-EQA.
• Agents storing unstructured episodic semantic memories (MemoryEQA) gain negligible accuracy (+1.4% MA) and sometimes experience retrieval noise as sequences grow.
• UniNavid, a VLA method relying on implicit short-term memory trained episodically, shows out-of-distribution degradation and negligible efficiency gains (-1.5% SA) when memory persists across tasks.
• Sequential memory reuse decouples navigation efficiency and answer accuracy for weak memory architectures, where shorter paths do not imply better reasoning.
• Spatial grounding of visual memory prevents catastrophic forgetting and semantic interference by anchoring observations in metric 3D geometry.
• Real-robot experiments corroborate simulation findings; 3D-Mem improves from 20% to 40% accuracy with sequential memory, while other architectures show marginal or negative gains.
• Most scenes in the OpenEQA benchmark contain 10-13 questions per environment, highlighting the need for long-horizon memory architectures.

Threat model

The adversary is the environment's sequential question complexity and temporal dependencies that test the agent’s ability to accumulate and reuse knowledge. The agent must operate continuously across queries without memory resets, confronting distributional shifts from extended interaction histories. There is no active adversarial attack or environment modification—only the natural challenge of long-horizon sequential embodied reasoning.

Methodology — deep read

1. Threat Model & Assumptions: The work assumes an embodied agent interacting in a 3D indoor environment, tasked with answering a sequence of natural language questions about the scene. The agent operates continuously, retaining internal state (memory) across queries. The adversary is the complexity and temporal dependencies in the environment and question sequence, not an active attacker. The agent has no ability to reset memory during the sequence.

2. Data: The authors use OpenEQA, a public dataset of real indoor scenes with free-form embodied question answering queries. They sequentialize the dataset by grouping questions by scene, creating query sequences averaging 10-13 queries each. No environment or question is modified; only evaluation protocol changes. The data size includes 72% of scenes containing at least 10 questions.

3. Architectures Evaluated: Four representative agents spanning two paradigms are examined:

• ExploreEQA: VLM-agent maintaining a 2D occupancy map with frontier scoring. Stores geometry but no semantic object info.
• MemoryEQA: VLM-agent building an episodic semantic library that stores RGB frames, pose, language descriptions, and embeddings as independent events.
• 3D-Mem: VLM-agent constructing a persistent 3D metric reconstruction with fused visual-semantic embeddings anchored to spatial coordinates.
• UniNavid: VLA approach using a fine-tuned transformer-based VLM with implicit latent memory over short episodic windows.

Input: Language question and visual RGB/depth observations. Output: Navigation actions and question answers.

4. Training & Adaptation: All models are trained in an episodic manner on short-horizon tasks with no sequential training or fine-tuning. Weights remain frozen during evaluation. A minimal memory adaptation protocol is applied: at the boundary between queries in a sequence, the final memory state of the prior query becomes the initial memory for the next query. No modification to weights, query inputs, or planners is made.

5. Evaluation Protocol: Two modes: Episodic (reset memory after each question) vs Sequential (carry memory forward). Metrics:

• Success Rate (SR) and SR with memory (SRmem) = mean accuracy.
• Memory Advantage (MA) = SRmem - SR.
• Path Length (PL) and PL with memory (PLmem).
• Step Advantage (SA) = normalized reduction in navigation cost.

Experiments run on A5000 GPUs with Qwen3 VLM 8B-Instruct foundation model.

6. Reproducibility: Code, sequential splits, and evaluation configurations are publicly released.

Concrete example end-to-end: For a scene with 3 questions, episodic evaluation resets the internal map each time, requiring full exploration and resulting in some incorrect answers (e.g., misidentified smoke detector color). Conversely, 3D-Mem under sequential evaluation accumulates a fused 3D spatial-semantic map, allowing direct navigation to relevant spots for later queries. This yields higher accuracy and dramatically reduced path length (~53% shorter). No retraining is performed; benefits arise solely due to the structured spatial memory enabling knowledge accumulation.

This step-by-step reveals that architectures with geometric spatial grounding naturally support sequential embodied reasoning, while unstructured episodic or purely geometric but semantics-light maps fail to retain reusable evidence for subsequent queries.

Technical innovations

• Sequential-EQA evaluation protocol converting episodic EQA benchmarks into continuous multi-query sequences to isolate memory reuse efficacy without retraining.
• Demonstration that memory persistence alone is insufficient—highlighting a critical architectural bottleneck in existing EQA agents' inability to accumulate reusable knowledge across queries.
• Empirical evidence that structured, spatially grounded 3D memory breaking the accuracy-efficiency tradeoff by fusing visual-semantic embeddings onto metric 3D geometry enables compositional knowledge accumulation.
• Minimal memory adaptation protocol—transferring terminal memory states between queries—enabling diagnostic separation of memory architecture effects from training biases.

Datasets

• OpenEQA — ~real indoor environments with free-form embodied questions — public

Baselines vs proposed

• ExploreEQA: SR = 43.8%, SRmem = 46.5%, MA = 2.7%, PL = 84.3m, PLmem = 84.3m, SA = 0.0%
• MemoryEQA: SR = 61.0%, SRmem = 62.4%, MA = 1.4%, PL = 43.6m, PLmem = 43.9m, SA = -0.5%
• 3D-Mem: SR = 25.5%, SRmem = 58.8%, MA = 33.3%, PL = 5.6m, PLmem = 2.6m, SA = 53.3%
• UniNavid: SR = 36.4%, SRmem = 37.3%, MA = 0.9%, PL = 12.2m, PLmem = 12.4m, SA = -1.5%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21571.

Fig 1: Comparison between Episodic and Sequential Evaluation paradigms. (a) In episodic evaluation, the agent’s memory is cleared

Fig 2: Distribution of questions per scene in the sequentialized

Fig 5: Qualitative comparison of agent trajectories under episodic

Fig 6: Real-world deployment of the Unitree Go2. In both

Fig 3: Accuracy–efficiency tradeoff under sequential memory reuse. From left to right: mean answer score (1–5; ↑), mean navigation time

Fig 4: Per-query-index accuracy and navigation time under sequential memory reuse. Panel A shows mean answer score (1–5; ↑, log

Limitations

• Sequential evaluation uses frozen episodic training weights; no specialized sequential training or memory consolidation methods were explored.
• Real-world robot tests have limited sample size and environment diversity, making generalization to more complex settings unclear.
• Memory persistence does not guarantee knowledge accumulation; this diagnostic protocol reveals failure modes but not how to design optimal sequential training.
• The approach assumes static scenes; dynamic environments and moving objects are not evaluated.
• Navigation planners are reset at each query start, which might limit holistic long-horizon trajectory planning.
• Only four representative architectures evaluated; other emerging memory forms might behave differently.

Open questions / follow-ons

• How can episodically trained agents be effectively fine-tuned or architected to leverage sequential memory via explicit continual learning or memory consolidation?
• What novel memory structures or retrieval algorithms best mitigate semantic interference and retrieval noise for long query sequences?
• Can these findings generalize to dynamic or interactive environments where scene contents change over time?
• How does sequential memory reuse impact robustness to noisy or partial observations beyond those tested?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this research underscores that persistent memory architectures in embodied agents are nontrivial to design correctly. Simply reusing past context or memory states without appropriate spatial grounding and structured fusion can degrade decision quality despite reducing navigation effort, analogous to how naive session persistence in challenging CAPTCHAs might not improve bot detection robustness. Incorporating spatially grounded, structured memory representations allows continuous agents to accumulate environmental knowledge compositionally, yielding efficiency gains without sacrificing accuracy. This principle applies broadly to any system requiring sequential interaction and memory reuse under real-world conditions.

From a design perspective, systems that maintain state across sessions or queries must carefully consider the memory format and retrieval mechanisms to avoid catastrophic forgetting and semantic drift. In CAPTCHA contexts, analogous architectural bottlenecks may occur when attempts to accumulate user interaction signals across sessions introduce noise rather than improve authentication accuracy. Overall, the paper advocates for diagnostic evaluation protocols transitioning from isolated to sequential assessment to reveal hidden memory system failure modes—an approach valuable to bot-defense engineers seeking scalable, continuous authentication solutions.

Cite

@article{arxiv2607_21571,
  title={ Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering },
  author={ Zikui Cai and Kaushal Janga and Tan Dat Dao and Seungjae Lee and Shivin Dass and Mingyo Seo and Kaiyu Yue and Mintong Kang and Nandhu Pillai and Monte Hoover and Aadi Palnitkar and Ruchit Rawal and Ruijie Zheng and Bo Li and Yuke Zhu and Roberto Martín-Martín and Tom Goldstein and Furong Huang },
  journal={arXiv preprint arXiv:2607.21571},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21571}
}

Read the full paper

• arXiv:2607.21571 (abstract)
• arXiv:2607.21571 (PDF)