A tool told someone to deploy a table from a stack they had deleted
Three bug reports arrived from one person running my analyzer against a CDK monorepo fronted by a single CloudFront distribution. The commit that fixed them describes the worst one in a line: stale cdk.out templates were being reported as live infrastructure.
That sounds mild. Here is what it actually produces. The shape is now a checked-in regression fixture, so I can show it with the fixture's names rather than anyone's real ones:
1. MED IaC drift: DynamoDB table "LegacyInvoicesTable" defined in IaC but not deployed
"LegacyInvoicesTable" is in cdk.out/LegacyBillingStack.template.json but not found
in AWS. It may be undeployed or deleted manually.
→ Run `terraform apply` / deploy your stack, or remove the definition from IaC.
That table is not missing. LegacyBillingStack was deleted from the app. The table is correctly absent from AWS, and the tool is recommending its return.
Worse than the wrong answer is the wrong answer's provenance. "It is in your IaC, at this file path" is the kind of statement that ends an argument. An assistant reading that finding has no reason to doubt it, and every reason to write code against a table that does not exist or to helpfully re-add the stack.
How a deleted stack keeps its template
cdk synth writes one CloudFormation template per stack into cdk.out, plus a manifest.json describing the app. Delete a stack from the app, or rename it, and the next synth writes the new set of templates. The template from the deleted stack is still sitting there from the last synth that included it.
Two properties of a monorepo turn that leftover file into a real problem.
cdk.out is a build directory, and the .gitignore that cdk init generates lists it. Nothing in git status, nothing in a code review diff, nothing in CI will ever show you a stale file in there. It exists only on developer machines and build agents, which is exactly where analysis tools run.
And in a monorepo you cannot see it by looking. On a single-stack app, an extra .template.json next to your one real one is noticeable. With a dozen stacks in one repo, and stacks getting split, merged, and renamed as the repo grows, one extra file among thirteen is invisible. The repo with the most stack churn is the repo where leftovers are hardest to spot.
The file itself gives you no help either. An orphaned template is valid CloudFormation. Parse it and you get a table name, a key schema, a billing mode, an SQS queue, an output with an Export.Name, all structurally identical to the live ones. Nothing inside the file says the stack that produced it no longer exists.
Why everything downstream believes it
Follow the orphan forward and every consumer inherits the error.
The drift analyzer compares what is declared in IaC against what was extracted from AWS. Every resource in the dead stack lands on the declared side and nothing on the deployed side, so each one becomes a "defined but not deployed" finding. A deleted stack does not produce one wrong finding, it produces one per resource, all medium severity, all confidently sourced to a real file path on disk.
The graph gets the same resources as nodes, so a question like "which tables exist in this service" is answered with a table that was decommissioned two quarters ago.
Cross-stack exports are the sharpest edge. The orphan's Outputs section still carries Export.Name: demo-legacy-invoices-table-arn, which reads as an export you can Fn::ImportValue against. That import compiles, synths, and fails at deploy time, because the export was removed from CloudFormation along with the stack.
And notice what the naive fix breaks. If you simply drop the orphan's outputs along with its resources, a dead cross-stack export becomes silently missing rather than visibly wrong, and the next person to go looking for demo-legacy-invoices-table-arn gets nothing back with no explanation. Those two cases need opposite treatment.
The file the build rewrites every run
cdk.out/manifest.json is the one artifact that knows what the app looks like today. cdk synth rewrites it every run, and it lists exactly the stacks the app instantiates, as aws:cloudformation:stack artifacts each pointing at their templateFile. That is true whatever language the CDK app is written in, because the manifest is the assembly contract, not a TypeScript detail.
So a *.template.json that no manifest artifact points at is an orphan from a deleted or renamed stack. That gives three grades of confidence, and they get three different treatments:
Not in the manifest. Its resources are excluded entirely: they never reach the graph, so they never reach the drift analyzer, and nothing downstream can act on them. Its outputs are kept and marked stale: true with the reason, so a dead export stays visible instead of vanishing.
In the manifest, but the file's mtime lags the newest template by more than an hour. Flagged, resources kept. Synth rewrites every template each run, so a lagging timestamp is real evidence of something, but a weaker kind, and excluding resources on it would throw away live infrastructure over a clock skew.
No readable manifest. No cross-check is possible, so nothing is flagged and nothing is excluded. Absence of evidence is not evidence.
That last rule is not a nicety. A tool that flags resources on a guess trains you to skim past its findings, and then it does not matter what it detects.
The exclusion that did not hold
Excluding orphans in the CDK parser was not enough, and the bug that followed is the kind you only find by testing the whole pipeline instead of the unit.
There is a second parser for hand-written CloudFormation templates, and it walks the repo recursively. It walked straight into cdk.out and re-admitted the orphaned stacks under a cloudformation source, so the resources the CDK path had just excluded came back in through the side door. The exclusion looked correct in isolation and did nothing in practice.
cdk.out is now on the recursive scan's excluded-directories list, alongside node_modules, .git, and dist, with a comment saying why: that directory belongs to the CDK parser, which cross-checks it. The regression test asserts on both entry points, the CloudFormation extractor and the combined one, because only the combined one catches this class of bug.
The remaining tests are the negative controls: resources from an unlisted template stay out, its outputs stay in with the flag, a lagging mtime raises the flag while keeping the resources, and with no manifest present every template is treated as current.
What changed in the loop
The drift report stopped inventing work. get_stack_outputs returns stale and staleReason per output, and the tool description tells the assistant in as many words not to rely on a stale export without re-synthesizing, so the model does not have to infer what the flag means.
The demo carries the broken shape on purpose as a checked-in fixture: a manifest.json listing one stack, and an orphan template next to it whose table and queue must not appear anywhere while its export must appear flagged. Delete the manifest and both stacks are treated as current again, which is the third rule made visible.
If you run a CDK monorepo, the check you can do right now without any tooling: list cdk.out/*.template.json, list the templateFile values in cdk.out/manifest.json, and diff them. Anything in the first list and not the second is a stack your app no longer builds, still describing infrastructure to anything that reads that directory.
Key takeaways
-
Build output outlives source. A deleted or renamed CDK stack leaves its synthesized template in
cdk.out, and because that directory is gitignored, no review or CI step will ever show it to you. -
Ground "what exists" in the file the build rewrites every run.
cdk.out/manifest.jsonlists the stacks the app instantiates today. The templates beside it only record what it once did. - Exclude dead resources, keep dead exports, flag them. They are opposite failures: a resource that no longer exists must not be acted on; an export that no longer exists must not disappear silently.
- Absence of evidence is not evidence. With no manifest to check against, the honest output is nothing at all, not a guess in either direction.
- Test the pipeline, not the unit. A correct exclusion in one parser was undone by a second parser recursing into the same directory. Only an end-to-end assertion caught it.
