# Google's sleep framework lets LLMs keep what they learn

> Google Research proposes that a model alternate between working and a sleep phase that writes short-term memory into its weights, with gains on math and knowledge tests.

URL: https://claudegpt.si/news/sleep-framework-llm-continual-learning  
Published: 2026-10-27T18:51:54.599Z  
Author: ClaudeGPT Si Newsroom  
Section: Research  
Topics: Google Research, Continual learning, Language models, Mixture of experts  
Publisher: ClaudeGPT Si

## Key takeaways

- Google Research proposed on October 27, 2026 a wake and sleep cycle in which an LLM moves short-term memory into its weights.
- With Qwen3-8B the paper reports 79.2 on AIME 24, against 76.6 for the best baseline and 73.8 for the base model.
- The paper has no limitations section, no linked code, and the results are the authors' own.

The sleep framework is a way for a language model to keep what it learns without forgetting what it already knew. Google Research described it in a [blog post on October 27, 2026](https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/), based on a [paper on arXiv](https://arxiv.org/abs/2606.03979) by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni. It is a proposal for continual learning in LLMs, and the team reports gains on math, long-context and knowledge tests.

The idea is borrowed from people. The model alternates between a wake phase, where it works and holds new information in short-term memory, and a sleep phase, where it stops taking input and writes that memory into its weights.

## Why LLMs can't just keep learning

A deployed model is mostly frozen. It knows what it learned before its training cutoff, and nothing after.

There are two obvious ways to teach it more, and both have a cost. The first is to update its weights as new data arrives. That tends to cause catastrophic forgetting: learning a new task degrades old ones. The second is to put the new facts in the prompt, which is called in-context learning. That works, but the model keeps the knowledge only while the text sits in its context window. When the session ends, it is gone.

Continual learning is the effort to get the benefits of both: learn from new data, keep the old skills, and keep the new knowledge.

## What the sleep framework does

According to the paper, the model's core architecture stays fixed. It uses a mixture-of-experts design, which splits the network into many small expert blocks and runs only some of them for each input. Some blocks are kept spare for future learning.

Sleep has two stages.

**Memory consolidation** moves short-term memory into long-term parameters. New low-rank experts, which are compact add-on blocks, are switched on and slotted into slower-changing memory. The paper calls the transfer "knowledge seeding": a form of distillation, where one model's behavior is used to train another, combined with reinforcement-learning-based imitation. In the paper's abstract, the distillation runs from a smaller version of the model into a larger one. After consolidation, the fast temporary parameters are reset.

**Dreaming** is for self-improvement. The model generates its own practice examples, with the router randomly activating unusual experts to add variety. It filters the examples, trains on the best ones, and checks whether performance improves.

The sleep framework builds on Hope, an earlier architecture from the same group ([arXiv](https://arxiv.org/abs/2512.24695)).

## What the results show

All numbers below are from the paper's tables. They are the authors' own results, not independent tests.

On math reasoning with Qwen3-8B, averaged over 16 runs, the sleep-trained model scored 79.2 on AIME 24, 69.0 on AIME 25 and 46.1 on HMMT 25. The best baseline on each was GRPO or OPSD, two other training methods, at 76.6, 68.1 and 45.1. The base model scored 73.8, 68.1 and 42.4. A smaller Qwen3-1.7B showed the same pattern.

On learning facts from a text passage (SQuAD), the four-level version scored 48.9 for a single passage and 46.2 when continually pre-trained on 200 passages. The base model scored 31.9 on both. SEAL, an earlier self-editing method, scored 46.7 and 43.2. Without the dreaming stage, the score fell to 35.7 and 36.2, which suggests dreaming does most of the work here.

On few-shot ARC puzzles with a Llama-3.2-1B model, sleep reached an 80% success rate against 72.5% for SEAL and 10% for test-time training. This test used 11 training tasks and 8 held-out ones, so it is small.

Google's post also reports near-perfect results on the BABILong long-context benchmark up to 10 million tokens, and almost no forgetting when translating Manchu and Kalamang, two languages a model is unlikely to know. In the paper those results appear in charts rather than tables, so we can't give exact figures. The class-incremental tests use Llama-3B and Llama3-8B models.

## What to keep in mind

The paper has no limitations section, so the caveats come from reading it.

- The gains on the math tests are a few points, from a model that was already strong.
- The efficiency picture is mixed. An appendix says that for the same number of training steps, plain fine-tuning is about four times more efficient than this method.
- Several headline comparisons are shown only in figures.
- Neither the post nor the paper text we read links to released code, so the results can't yet be reproduced by outsiders.
- The paper is a preprint. We found no independent replication.

## Why it matters

If a model could consolidate what it learns after deployment, it would not need a full retraining run to learn a new language, a company's documents or a changed fact. That is the practical prize behind continual learning, and no lab has delivered it at scale. This paper is a credible step with decent numbers on small and mid-size models. It does not show that it works on frontier models, or over months of real use.

What to watch: whether Google releases code or builds the approach into a product, and whether other groups reproduce the SQuAD and math gains.

## Frequently asked questions

### What is continual learning in LLMs?

It is the effort to let a model keep learning from new data after deployment without forgetting what it already knows. Plain weight updates cause catastrophic forgetting, and knowledge in the prompt is lost when the session ends.

### What is catastrophic forgetting?

It is when training a model on a new task degrades its performance on tasks it learned earlier. It is the main obstacle to updating a deployed model continuously.

### Is the sleep framework available to use?

Not as far as the post and the paper text we read show. Neither links to released code, and Google has not announced a product that uses it.

## Sources

- [The sleep framework: A new way for LLMs to retain knowledge in continual learning](https://research.google/blog/the-sleep-framework-a-new-way-for-llms-to-retain-knowledge-in-continual-learning/), Google Research
- [Sleep paradigm paper (arXiv 2606.03979)](https://arxiv.org/abs/2606.03979), arXiv
- [Hope architecture paper (arXiv 2512.24695)](https://arxiv.org/abs/2512.24695), arXiv