The chat history can survive when you switch AI models in your IDE, but that does not mean the workflow survives with it. Here is what actually carries over, what does not, and why agent state matters more than a giant transcript
I have been using AI coding agents enough lately that switching models inside an IDE has started to bother me more than I expected.
Most of these tools make changing models look almost meaningless. You are halfway through a conversation, click a dropdown, choose another model, and keep working. The chat is still sitting there, your files obviously did not disappear, the repository is in the same state, and the replacement model may even have access to everything that was said before it arrived.
For a while, I treated that like swapping the engine behind the same agent. Maybe one model was faster, another was better at reasoning, I ran into a usage limit, or a newer model showed up that supposedly performed better at coding. The IDE already knew what we were doing, so I assumed the replacement would basically pick up the same job.
After spending much longer sessions working this way, I do not think that is what is actually happening.
The information can survive a model switch while the way the work is being done changes almost immediately.
That sounds like a small distinction until you have spent three or four hours getting an agent into a workflow you actually like, switch models, and suddenly find yourself wondering why it is touching files the previous one had learned to leave alone.
The model dropdown makes this look simpler than it is
This is what originally got me thinking about the problem.
Modern IDEs can put a whole list of models behind one little dropdown. VS Code does exactly that now, including built-in models, automatic selection, and models added from other providers.
That interface naturally makes the models feel interchangeable.
The conversation is above the dropdown.
The model is below it.
Change the model, keep the conversation.
Simple.
Except the conversation is only one part of what made the previous agent useful.
Suppose I have spent the afternoon on a repository with Model A. During that time it has searched through the project, changed code, broken something, traced the failure, read test output, discovered a weird dependency, learned that one ugly-looking file is intentionally being left alone, had a couple ideas rejected by me, and eventually settled into a way of approaching changes that works well for this particular project.
Then I switch to Model B.
The IDE may be perfectly capable of giving Model B the previous conversation.
Model B still did not spend the afternoon doing any of that.
It is reading the evidence left behind by the previous work and deciding for itself what that evidence means.
That is a different thing.
The IDE remembers. The model interprets.
This became much easier for me to understand once I stopped thinking of the "AI in my IDE" as one thing.
There is an agent harness, and there is a model.
The harness can maintain the session around the model. It can keep the conversation, inspect the repository, include project instructions, attach files, collect terminal output, store tool results, track source-control changes, and decide what information belongs in the next request.
The selected LLM gets whatever context that harness assembles and then has to make sense of it.
VS Code's documentation actually has a really useful diagram for this:
That is a much better mental model than imagining one persistent AI brain sitting inside the editor.
The flow is closer to:
Project
Conversation
Instructions
Git state
Files
Tests
Tool results
|
|
v
Agent harness
|
| builds the current context
|
v
Selected model
|
| interprets that context
|
v
Decision / response / tool call
If I replace the model, I have not necessarily thrown away the upper half.
What I replaced is the thing at the bottom deciding what all of it means.
That explains a lot of behavior I used to describe as the model "forgetting."
Sometimes it did not forget anything.
It just came to a different conclusion.
Two models can read the exact same project instructions and behave completely differently
Imagine I have an AGENTS.md file containing this:
## Development rules
- Inspect the existing implementation before editing.
- Preserve the current architecture.
- Prefer small changes over large rewrites.
- Run relevant tests after meaningful changes.
- Do not remove working functionality just to make a test pass.
- Read the existing documentation before changing a subsystem.
Those are useful instructions. I would much rather have them than rely on the model figuring out my preferences every time.
They also leave a ridiculous amount open to interpretation.
What exactly counts as a small change?
How much of the repository should be inspected before the model understands the implementation?
What does preserve the architecture mean when the model decides the architecture is causing the problem?
Which tests are relevant?
Is creating another abstraction a small change because it only touches two files, or a large change because it changes how future code should be structured?
One model may read those instructions, inspect three files, patch one function, run the closest tests, and stop.
Another may spend twenty minutes exploring the repository because it does not think it has enough information yet.
A third might immediately recognize the problem, modify one line, and be completely right.
Another could see a slightly ugly implementation and start "cleaning it up" even though I would rather leave the ugly working code alone.
None of those models necessarily failed to receive the instructions.
They interpreted them differently.
That is the part that cannot be solved by saying:
Just give every model the same prompt.
The prompt is not the decision.
It is input to a decision.
Good agent notes still make a massive difference
None of this means documentation is pointless. I actually think the opposite.
The more I switch models, the more useful good project notes become.
There is a huge difference between handing the next model this:
Continue working on retrieval.
and handing it this:
## Current retrieval work
Standalone BM25 retrieval is currently performing well.
Do not replace the retrieval pipeline.
We already tested applying Reciprocal Rank Fusion globally and
rejected that approach because it reduced ranking quality on
standalone queries.
RRF is currently reserved for contextual follow-up questions where
multiple query views provide meaningfully different rankings.
### Current goal
Improve follow-up question handling without hurting standalone retrieval.
### Already tested
- Global RRF
- Larger retrieval sets
- Additional correlated query variants
### Why those approaches were rejected
- Global RRF reduced standalone ranking quality.
- Larger retrieval sets increased prompt size without improving answers.
- Correlated ranking lists added noise without enough new information.
### Likely next area
Improve contextual query rewriting before retrieval.
Now the replacement model knows much more than the current code can tell it.
It knows why the architecture looks the way it does.
It knows an idea that sounds reasonable has already been tested.
It knows the current goal is not "make retrieval better somehow." There is a specific boundary around the work.
That could easily prevent an hour of duplicated effort.
What it cannot do is make Model B become Model A.
Model A might look at those notes and make a tiny change inside the query-rewriting code.
Model B might decide the rewriting layer itself is the real problem and should be reorganized.
Another model might refuse to change anything until it reruns the evaluation suite because it does not trust the previous measurement.
Same repository.
Same notes.
Same task.
Different interpretation.
The notes preserve the history of the engineering decisions. They do not preserve the exact engineer that made them.
What feels lost is usually harder to write down
The weirdest model switches are the ones where the replacement clearly knows what is going on.
It can explain the architecture correctly.
It knows which files were changed.
It understands what I am trying to build.
It can even repeat my instructions back to me.
Then it starts working and I immediately think:
Why are you doing it like this now?
That is the part I used to have trouble explaining.
A long coding session creates a lot of context that does not fit neatly into "facts."
Maybe I rejected three large refactors earlier, so the first model gradually became more conservative about changing working code.
Maybe I kept telling it to stop getting stuck investigating one small issue once we had enough information to continue.
Maybe it learned from my reactions that I would rather run the real application and see what breaks than spend another fifteen minutes theorizing about what could break.
Maybe we attempted a clever abstraction, it caused regressions everywhere, reverted it, and then continued using the simpler implementation.
Maybe there is a file that looks obsolete but the model discovered two hours ago that some old integration still depends on it.
Every one of those things can theoretically be documented.
In a real session, they are not always documented.
A lot of them appear once inside a conversation, a terminal result, or an investigation that seemed minor at the time.
The first model had those events sitting in the context surrounding its work. A replacement model may get the same history, but now it has to independently decide which of those hundreds of details are important.
It does not have to rank them the same way.
A transcript is a pretty terrible save file
This is the part where being someone who likes video games actually gave me a better way to think about AI memory.
Imagine playing an RPG for forty hours and the game saved your progress as this:
Brad entered the village.
Brad talked to the merchant.
Brad purchased a sword.
Brad entered the forest.
Brad fought two enemies.
Brad found a key.
Brad returned to the village.
Brad sold the sword.
Brad purchased an axe.
Brad entered the cave.
Brad opened a chest.
...
You could probably reconstruct a decent amount of the player's state by reading all of that.
It would still be an insane save system.
Games store the current state directly:
{
"level": 18,
"health": 87,
"currentQuest": "black_tower",
"questStage": 4,
"inventory": [
"iron_axe",
"tower_key"
],
"completedQuests": [
"lost_merchant"
],
"unlockedAreas": [
"north_pass"
],
"decisions": {
"spared_guard": true
}
}
I think coding agents need more of that mentality.
Instead of depending on:
Here are 80,000 tokens describing everything we did today.
Please figure out which parts matter.
I would rather have something resembling:
{
"goal": "Improve contextual follow-up retrieval",
"currentStrategy": {
"standalone": "Direct BM25",
"contextual": "Rewritten query views with conditional RRF"
},
"constraints": [
"Do not reduce standalone retrieval quality",
"Do not significantly increase the normal prompt budget"
],
"rejectedApproaches": [
{
"approach": "Global RRF",
"reason": "Reduced standalone MRR"
},
{
"approach": "Increase top-k globally",
"reason": "Increased prompt size without improving answer quality"
}
],
"changedFiles": [
"lib/query-understanding.js",
"lib/conversation-resolver.js"
],
"verification": {
"retrievalSuite": "passing",
"conversationSuite": "2 scenarios failing"
},
"knownFailures": [
"Ambiguous pronoun after project comparison"
],
"nextLikelyAction": "Improve active-entity resolution before retrieval"
}
That is not a replacement for conversation history.
It is a way of promoting the important parts of the conversation into actual state.
Now the next model does not have to reconstruct the entire engineering process from a transcript every time it enters the project.
It gets something much closer to a save game.
Long context has its own problems
There is another reason I do not think "just keep the whole conversation" is enough.
Context windows are finite, and IDEs have to decide how much information belongs in them.
VS Code exposes this directly now. The session interface can show how much of the model's available context is being consumed by system instructions, tools, user messages, tool results, and the rest of the session.
At some point a long conversation has to be trimmed, compacted, summarized, retrieved selectively, or otherwise reduced.
That immediately creates another engineering problem:
What gets kept?
Suppose a three-hour authentication debugging session becomes this summary:
The user is modifying authentication.
Refresh-token rotation has been fixed.
Current tests pass.
The user prefers incremental changes and does not want
the authentication architecture rewritten.
Current task is adding session revocation.
That seems like a pretty useful summary.
Unfortunately, during the actual investigation the first model may have discovered this:
auth/session.js appears obsolete, but the mobile client still
uses its legacy endpoint. Do not remove it yet.
If that fact never became durable project documentation and the summarization process leaves it out, the replacement model might remove the file for completely reasonable reasons.
Technically, nobody "forgot the entire conversation."
One important fact simply did not survive compression.
That distinction matters when we are trying to build reliable agents instead of chatbots that merely feel continuous.
Some discoveries should stop being conversation
This is where I think the project itself needs to absorb knowledge from the agent.
If a model discovers something important, I do not want that discovery to live forever inside turn 73 of some chat session.
Depending on what it learned, the result might belong in:
- a regression test
- an architecture decision record
- a code comment
- project documentation
- a task-state file
- an issue
- a schema
- a validation rule
- a benchmark
- the git history
Once that happens, the information no longer depends on the original model being around.
That is a much stronger form of memory.
Tests are probably one of the best forms of cross-model memory we already have
Suppose Model A discovers a subtle bug in how conversation context affects standalone searches and adds this:
it('does not rewrite standalone queries using stale conversation entities', () => {
// regression test
});
Model B does not need to understand every argument and failed experiment that eventually produced that test.
It can implement the next feature however it wants.
If it breaks the behavior, the test fails.
The lesson survived because it stopped being:
Remember that thing we talked about earlier.
and became:
The repository now enforces this behavior.
That same idea applies to type systems, schemas, architecture checks, API contracts, linters, regression fixtures, CI rules, and evaluation suites.
Every time I can turn:
The previous model learned not to do this.
into:
The software can detect when someone does this.
I have made the project less dependent on conversational memory.
That is a much stronger guarantee than adding another paragraph to an instruction file.
Git is memory too
Git is another part of this that I did not appreciate enough until I started relying heavily on coding agents.
An incoming model can inspect:
git status
git diff
git log
and recover a surprising amount of useful working state without me explaining anything.
It can see which files changed.
It can compare the current implementation to the last commit.
It can find when an architectural decision entered the codebase.
More importantly, a useful commit message can preserve why something was changed.
For example:
fix: keep RRF off standalone retrieval
Testing global RRF lowered MRR on the golden retrieval set.
RRF should only combine rankings for contextual follow-ups
where conversation history creates meaningfully different
query views.
That decision no longer exists only because one model happened to remember the experiment.
It is part of the project history.
A different model can find it next week.
Another developer can find it six months later.
I can find it after I completely forget why I wrote the code that way.
That is actual durable memory.
Multi-model routing has the exact same problem
This becomes even more important when the system is intentionally designed to use several models.
The basic idea sounds great.
Use a fast model for simple work.
Send architecture questions to a stronger reasoning model.
Use another model for review.
Switch providers if one is down.
Fall back to something cheaper when the expensive model is not necessary.
I like that architecture.
What gets ignored surprisingly often is the handoff.
Imagine Model A spends twenty minutes debugging latency and eventually establishes this:
Observed symptom:
Requests intermittently exceed the latency budget.
Ruled out:
- Network path
- Prompt length
- Prompt cache behavior
Measured:
Concurrent inference causes roughly a 10x slowdown.
Current hypothesis:
A background inference path overlaps user generation.
Next investigation:
Find every code path capable of invoking inference concurrently.
Then the router replaces it with Model B and sends:
The user is debugging latency. Continue.
That is not intelligent model routing.
That is throwing away the investigation and hoping the next model rediscovers it cheaply.
A real multi-model agent has to route useful state along with the request.
Otherwise every handoff is basically hiring a new engineer and giving them half of the previous engineer's sticky notes.
The handoff I actually want is boring, and that is probably good
The more I think about this, the less magical my ideal model handoff looks.
Something like this would be far more useful to me:
MODEL HANDOFF
────────────────────────────────────────────
GOAL
Implement session revocation without changing the public
authentication API.
CURRENT STATE
Refresh-token rotation works.
Existing authentication tests pass.
Revocation storage has not been implemented.
ARCHITECTURE DECISIONS
JWT validation remains stateless.
Revocation state belongs in the session layer.
Do not replace the existing token service.
REJECTED APPROACHES
Database lookup on every authenticated request.
Reason:
It removes the existing stateless fast path and adds a lookup
to every normal authenticated request.
FILES CURRENTLY RELEVANT
src/auth/token-service.ts
src/auth/session-service.ts
test/auth/session.test.ts
CHANGES ALREADY MADE
Session ID added to refresh-token metadata.
No public API changes.
KNOWN FAILURE
A revoked refresh token can currently obtain one final access token.
NEXT STEP
Add revocation validation to the refresh flow and create a
regression test for the known failure.
PROJECT PREFERENCES
Prefer incremental patches.
Do not refactor unrelated authentication code.
Verify behavior before expanding scope.
CURRENT VERIFICATION
npm test: PASS
npm run typecheck: PASS
git working tree: 2 modified files
Give that to another capable model along with the repository, the relevant test results, and access to the previous conversation when it needs more detail.
Now I have an actual handoff.
It is not perfect continuity, but it is engineered continuity.
I trust that much more than leaving the old messages visible and assuming the next model will interpret all of them exactly the same way.
I no longer think this is mainly a "conversation context" problem
The question that originally sent me down this rabbit hole was simple:
If I switch LLMs in my IDE, am I losing the conversational context?
The technically correct answer can easily be no.
An IDE can absolutely keep the conversation and send it to the replacement model.
That is not the question I care about anymore.
What I really want to know is:
How much of the working state survives in a form the replacement model can use correctly?
Those are very different measurements.
I could preserve every message from an eight-hour session while burying the one architecture decision that actually matters somewhere inside 70,000 tokens.
I could also discard most of that transcript while explicitly preserving the current goal, constraints, rejected approaches, files involved, tests, failures, and next action.
I would probably trust the second handoff more.
So I do not really want infinite conversation history.
I want better state management.
This also changes how I think about model benchmarks
Most coding-model comparisons ask something like:
Which model solves more coding problems?
That is useful, but agent systems create another problem that I would love to see measured more often.
Start a real development task with Model A.
Let it explore the repository, run tools, make a couple mistakes, fix them, reject an approach, modify code, and get halfway through the job.
Then freeze the state.
Hand that same state to Models B, C, and D.
Now measure:
- Did they preserve the existing constraints?
- Did they repeat an approach that had already failed?
- Did they understand why the current architecture looks the way it does?
- Did they identify the right next action?
- How much code did they have to reread?
- How many tool calls were spent rediscovering previous work?
- Did they break an established behavior?
- How often did a person have to explain something that had already been established?
That would tell me something about the portability of the agent system.
A model could be incredible when handed a clean benchmark prompt and terrible at joining an ugly, half-finished real project halfway through.
Those are different abilities.
The model still matters, obviously
There is also a trap on the opposite side of this argument.
Once you start thinking about harnesses, retrieval, memory, tests, validation, structured state, and everything surrounding the LLM, it becomes tempting to say:
The model does not matter. The harness is everything.
I do not believe that either.
Different models absolutely behave differently.
One may be much more willing to refactor.
Another may follow instructions extremely literally.
One may be better at exploring unfamiliar code while another works better after the problem has been tightly scoped.
They can differ in context limits, tool usage, coding style, reasoning ability, verbosity, how cautious they are, and how quickly they decide they have enough information to make a change.
The harness can make those differences less dangerous.
It cannot remove them.
Replacing the model should not destroy the entire system, but it is still replacing an important component of the system.
"Swappable models" means more than changing an API endpoint
This matters a lot to me because model availability is not permanent.
Providers change limits.
Pricing changes.
Free tiers change.
Models get deprecated.
New ones appear.
The model that works perfectly for a project today may not be the one I want to use next year.
So I absolutely want a real model boundary in anything I plan to keep around.
But having this:
provider = "model-a";
and changing it to:
provider = "model-b";
does not automatically mean I built a portable agent.
If doing that causes the new agent to repeat rejected experiments, lose project conventions, misunderstand why code exists, or completely change the way the repository is being handled, I only built portable inference.
The agent itself still depended on the old model.
True portability requires enough intelligence to exist outside the current LLM that another one can walk in and understand where the project is without reconstructing the entire history from scratch.
This is part of what building Scout has been teaching me
I ran into a version of the same problem from the other direction while building Scout.
My earlier instinct was much closer to the usual AI architecture: give the model a bunch of information and let it figure everything out.
The longer I work on it, the more things I want moved out of that giant fuzzy responsibility.
If a fact matters, I would rather have a real source of truth.
If a relationship between two facts matters, I would rather validate the relationship.
If conversation state matters, I would rather track the relevant entities and decisions than blindly resend an unlimited transcript.
If the assistant absolutely cannot make a certain claim, I would rather enforce that outside the model than write:
PLEASE DO NOT HALLUCINATE
for the fourteenth time and hope capitalization finally becomes an architecture.
If a failure teaches me something important, I want that failure to become a regression test.
None of this gets rid of the LLM.
It makes the LLM responsible for the part it is actually useful for while giving the rest of the system something more durable than hoping one model keeps interpreting yesterday's conversation the same way forever.
Coding agents have the same problem. The domain is just much larger and much messier.
So yes, switching models can absolutely change the workflow
If I have spent hours working with one model and it has settled into a way of approaching the project that I like, I now expect the workflow to change at least somewhat when I replace it.
That can happen even when the chat history survives.
Good instructions reduce the difference.
Agent notes reduce it further.
Structured task state is even better.
Tests, git history, evaluation, documentation, architecture decisions, and validation give the next model things it does not have to infer from my conversation.
None of them can force two different models to interpret every situation identically, and I do not think that should be the goal anyway.
What I actually want is much simpler:
Changing the model should not mean changing the project's memory.
The replacement should be able to recover the goal, current state, important decisions, rejected approaches, constraints, evidence, known failures, and verification status without depending on one particular LLM having personally participated in the earlier conversation.
After that, the new model is free to reason differently.
That is the point of having different models in the first place.
I just do not want every model change to turn into the first day at a new job.
The conversation may survive the switch.
The exact workflow probably will not.
The engineering problem is making sure everything that actually matters survives anyway.
Sources and further reading
The screenshots and the parts of this article describing current VS Code behavior come directly from Microsoft's documentation:
