When Switching Models Turns Your Assistant Into a Stranger
You've probably had this experience: a new model comes out, you switch right away, and suddenly your AI assistant starts misreading months of notes you'd carefully built up. The vendor-specific caveats, the conclusions from last quarter's proposal, the customer-service rules you painstakingly wrote down — all of it is still sitting there, untouched, yet the answers come out wrong. Not a single line has been deleted, but the assistant behaves as if it's lost its memory.
For a solo business owner, this is no small problem. The assistant's notes are the business's working record, and rebuilding that record takes months. Swapping models has become a routine event that recurs every few months, yet migrating an AI agent's memory remains a gap nobody is minding.
The Study
A paper by Ankit Goyal and Jaideep Ray examines that gap through a controlled experiment. The researchers asked a single question: if you keep the same memory store and swap out only the model, what does the agent forget, and how?
They identify three routes to forgetting: the new model interpreting old notes differently; embeddings from different versions getting mixed into a single index and breaking retrieval; and recovery failing outright because the original source material is gone.
The experiment stores the same history in four different formats and compares them: keeping the raw text and reading it as long context (LC-RAW); chunking it for retrieval-augmented generation (RAG); having the model compress it into natural-language notes (NOTES); and normalizing it into a knowledge graph with a fixed schema (KG-fixed). The material was 48 synthetic histories, with answer keys assigned at random and scored strictly on exact correctness. The researchers used two open-weight models, each under 10 billion parameters.
What They Found
The results diverged sharply by storage format. Here's an at-a-glance look at the three formats for which the researchers reported migration scores as hard numbers.
The fixed schema barely budged, the summary notes turned out to be tightly bound to the model that wrote them, and the retrieval format lost most of its gains the moment only half the embeddings were refreshed.
The researchers went a step further and broke down where the losses came from. For summary notes, 80% of the accuracy shortfall traced back to information dropped at the moment the notes were first written. For retrieval, by contrast, 81% of the shortfall came from retrieval failures. In other words, the two formats forget for entirely different reasons.
The recovery experiment is also worth noting. Trying to fix summary notes using only the note store itself failed to hit the 90%-recovery target in all 48 cases. But when the original history had been kept alongside the notes, 34 of the 48 cases hit the target in one migration direction.
What to Do About It
The first move is simple: don't delete the source material. Even if the notes your assistant summarized look clean and tidy, keep the underlying conversation logs and documents on file separately. If the assistant starts acting strange after a model swap, the fix is to regenerate the notes from the source, not patch the notes themselves — and without the source, you have no way to fix anything.
The second is to run a small test before you commit to the switch. As this study shows, memory migration plays out differently depending on the direction — old model to new, or vice versa. Pick around twenty frequently asked questions, write down the current answers, then ask the same questions again after switching models. The number of answers that come out wrong is your signal for whether the switch is safe to make.
Third, if you're running a retrieval-based assistant, don't re-embed only half your index. Mixing the old index and the new one in the same store as a shortcut costs you most of the upside. Either rebuild the whole index from scratch, or keep using the old one as-is until the new one is fully ready.
Finally, the longer you plan to keep a piece of memory, the safer it is to store it in a fixed structure. For information with clear fields — vendor names, terms, dates — a table or a set template holds up better across model swaps than a freeform sentence, because how it's read doesn't shift with the model.
Caveats
This study was run on 48 synthetic histories, scored purely on exact-match correctness. Whether real-world business records — with all their tangled context — would show the same magnitude of difference still needs to be checked separately. The models were also limited to two open-weight systems under 10 billion parameters each, so the findings don't translate directly to larger models or commercial services.
The recovery figures, too, come from just one migration direction; the paper's reported results alone can't tell us whether the source material would carry the same weight in the opposite direction. Even so, the three principles — keep the source, test each direction separately, and never mix embedding spaces — read as practical guidance you can apply every time you migrate memory, regardless of scale.
