Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models
Abstract
World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.
Community
More memory isn’t enough—world models need the right memory at the right time. Most retrieval methods rely on fixed heuristics such as recency, pose proximity, or visual similarity, but the most similar past observation is not always the one that best predicts what happens next. FAR instead learns which memories are predictively useful, and which retrieval cues to trust for each query. It beats hand-designed retrieval even with the same cues, while remaining future-blind at inference.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory (2026)
- HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models (2026)
- Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision (2026)
- LOCI: Spatial Linear Memory for Streaming World Models (2026)
- LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation (2026)
- Learning What to Remember: Long-horizon Counterfactual Memory Optimization (2026)
- T$^2$Mem: Learning Test-Time Memory for Robotics (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2609.34677 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 3
1202kbs/FAR-Datasets-AI2THOR
1202kbs/FAR-Datasets-LoopNav
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper