I spent today on a control plane that provisions plain VMs over SSH — the kind of app where a wrong answer doesn't render badly, it reinstalls a database daemon on a machine that's serving traffic.
Here's the thing I had to admit: the only question the platform could actually answer about a host was "is there a row in my database with this address?"
That question is true of three completely different machines:
- A machine we bootstrapped and still administer.
- A machine somebody rebuilt on the same address — same IP, brand new disk.
- A machine that belongs to a different control plane entirely.
Those need three different answers, and the database can't tell them apart. Only the machine can.
The manifest: the machine's half of its own identity
So the bootstrap pipeline now writes a small JSON file onto every node it touches — call it /opt/control-plane/node.json. It's the machine's side of a two-sided identity: the database says "I believe node X lives at this address", the file says "I am node X, administered by this control plane".
Two design calls in that file are worth more than the file itself.
Nothing secret goes in it. It's world-readable on purpose — an operator SSH'd into that box should be able to answer "what is this machine?" without sudo. So: no private key, no password, no control-plane token.
And not the SSH host key fingerprint either, which is the one people reach for. That fingerprint is trust state on the control plane's side, about the machine. Writing it onto the machine would let anything that can edit the file choose what it will be compared against. That's not a fingerprint any more, it's a suggestion.
final readonly class NodeManifest
{
public function __construct(
public string $nodeUuid,
public string $providerUuid,
public string $controlPlane,
public string $stackVersion,
public string $agentVersion,
/** @var array<string, string> */
public array $components = [],
public ?string $bootstrappedAt = null,
) {}
public static function fromJson(?string $json): ?self
{
// Null covers "no file" and "a file we can't read as ours".
// The caller must treat both the same way: NOT RECOGNISED.
// What it must never do is treat either as recognised.
// ...
}
public function belongsTo(string $controlPlane): bool
{
return $this->controlPlane !== ''
&& rtrim($this->controlPlane, '/') === rtrim($controlPlane, '/');
}
}
A corrupt manifest is not an absent one — but it's equally not a match. Returning null for both is what makes the caller surface it instead of quietly reinstalling over a live node.
The probe: one SSH exec, before anything changes
Ahead of the pipeline, from every entry point that can start one, there's now a single read-only probe. It reads the manifest, checks the database, and returns an outcome enum.
enum NodeInventoryOutcome: string
{
case Bootstrap = 'bootstrap';
case Reconnect = 'reconnect';
case Upgrade = 'upgrade';
case RefuseUnknownNode = 'refuse_unknown_node';
case RefuseAddressClaimed = 'refuse_address_claimed';
case RefuseForeignControlPlane = 'refuse_foreign_control_plane';
/**
* One predicate, rather than three call sites each remembering
* which cases are refusals. A list that has to be kept in step
* with an enum is a list that stops being in step.
*/
public function mayProceed(): bool
{
return match ($this) {
self::Bootstrap, self::Reconnect, self::Upgrade => true,
self::RefuseUnknownNode,
self::RefuseAddressClaimed,
self::RefuseForeignControlPlane => false,
};
}
}
Three of six cases are refusals, and the refusals are the reason the mechanism exists. Unknown never resolves to proceed. Same rule as "unscanned doesn't mean clean".
The refusals need different messages, not one generic one
This is the part I'd have got lazy about a year ago.
- Address claimed — a row says node X is here, but the machine answering carries no manifest. It was rebuilt, or the address got recycled. The fix is retire the node record.
- Unknown node — the machine carries our manifest naming a node we have no record of. The fix is go find out where that record went.
Telling an operator "unknown node" in the first case sends them hunting for a record that exists. Two failures, two fixes, two messages. A refusal is an API — someone reads it and acts on it, and if every refusal says the same thing you've built a shrug.
Order matters too. The foreign-control-plane check runs before the uuid lookup, because "this belongs to someone else" is a more specific and more urgent answer than "we have no record of that uuid" — which would also be true of it. Two control planes rotating credentials on one host is how both lose it.
it('refuses an address whose machine carries no manifest', function () {
$node = ManagedNode::factory()->for($provider)->create([
'ssh_host' => '203.0.113.10', 'ssh_port' => 22,
]);
$result = probeWithManifest(null)->probe($provider, '203.0.113.10');
expect($result->outcome)->toBe(NodeInventoryOutcome::RefuseAddressClaimed)
->and($result->outcome->mayProceed())->toBeFalse()
->and($result->reason)->toContain('rebuilt');
});
The ledger: an upgrade should be a diff, not a reinstall
Second half of the day. Before this, the only tool for "make this node current" was re-run all fourteen bootstrap steps. On a machine already serving traffic, that's a reinstall nobody asked for.
So each step that installs something now declares what it puts on the machine and at what revision:
interface ContributesToNodeStack
{
/** Stable key — outlives the class name. */
public static function stackComponent(): string;
/** Bump in the same change as whatever alters what the step does. */
public static function stackComponentVersion(): string;
}
Declared on the step, not in a central list. The failure mode of a hand-kept list is an upgrade that silently skips the one thing that changed. The desired stack is walked out of the pipeline itself, so a step can't drift from the ledger that decides whether a machine needs it.
The methods are static because a fleet view answering "how many nodes are behind?" has no node in front of it — let alone an agent to reach one with.
Two rules keep the diff honest:
Absent counts as behind. A component the node never reported isn't "probably fine". The entire reason to record versions is that silence and currency must not look the same.
Intent and report are different facts and must never be written from each other. The constant in code is what the platform intends a node to carry. The column on the node row is what the machine reported. Deriving the second from "a job finished successfully" tells you a run completed — not what's on the disk now. A host somebody rebuilt by hand is precisely where those two diverge, and it's the case the whole mechanism exists for.
Ordering comes from the pipeline, not from the diff, because the chain encodes real dependencies — swap has to exist before the package manager can run at all on a small node.
The case I deliberately didn't wire up
Upgrade is in that enum. Nothing emits it yet. A node that's behind is reported as behind and still reconnects, because the runner that would act on an upgrade outcome isn't built.
That's on purpose. An outcome nothing can execute is a declared capability that never runs, and I've been pulling those out of this codebase all month. The case ships with its runner.
Takeaway
If your platform mutates machines, the row is not the machine. Get the machine to state its own identity, probe before you act, and make sure the answer "I don't recognise this" has exactly one legal consequence: stop.
The expensive version of this lesson is a fourteen-step pipeline running against someone else's production database.
