Short answer: keep one standby API credential provisioned, test it from a dark Node.js path, and switch by a short-lived feature flag; never create the replacement during the incident. The spend ceiling is the number of retained credentials and audit records. The dangerous alternative is refused traffic while a key-creation call waits on the same control plane that is already under pressure.
This is a healthtech problem, so “just retry” is not an incident plan. A medication reminder or lab-result notification can be delayed, but an authentication callback can also be part of a patient workflow. I design the rotation path as a small state machine with an explicit owner, an expiry time, and a rollback decision. It sounds fussy. It saves a late-night guess.
Count the bill before choosing a standby
The recurring cost is usually retention, not the one-time rotation request. Count each active credential, its secret-store version, replicated audit events, and the traffic you keep sending to a secondary path for health checks. A spare key that is stored but never exercised has a lower request bill and a higher failure risk. A spare key that receives full duplicate traffic has a lower discovery risk and a larger spend ceiling.
For a small service, one active key plus one standby is a reasonable starting policy. Keep the standby disabled at the application layer, but valid at the provider layer. If the provider charges per credential, include both in the budget. If it charges by request, cap the probe to a fixed rate and alert on that counter. Do not turn price into the availability argument; a refused request during an incident is the real operational cost.
The retention decision is deliberately asymmetric. We keep two current secret versions for the rotation window, then destroy the old value after logs and downstream caches have aged out. That means a forensic replay after the window may lose the original credential, which is acceptable only if the audit event retains who rotated it, why, and which incident ticket authorized it.
| State | Credential allowed to serve | What to measure |
|---|---|---|
| Active | Primary only | accepted requests, 401/403 rate |
| Dark standby | No normal traffic; probe only | probe success, latency, expiry |
| Cutover | Standby first, primary drain | refused traffic, retry count |
| Retired | Neither | revocation confirmation, audit event |
That table is also a spend control. It makes “spare” a bounded state instead of an ever-growing pile of credentials.
What should a standby API credential rotation failover do in Node.js?
The application should resolve a credential through an indirection layer, not read API_KEY in every module. In Node.js, that can be a small provider object backed by a secret manager and a process-local cache. The cache needs a monotonic refresh deadline; a wall-clock jump should not make the service rotate twice.
Here is a provider-neutral sketch of the decision logic. It is Python so the state transitions are easy to inspect; the same contract can sit behind a Node.js module.
from dataclasses import dataclass
from time import monotonic
@dataclass
class Credential:
value: str
expires_at: float
class CredentialSet:
def __init__(self, primary, standby):
self.primary = primary
self.standby = standby
self.cutover = False
def serving(self):
candidate = self.standby if self.cutover else self.primary
if monotonic() >= candidate.expires_at:
raise RuntimeError("credential_expired")
return candidate.value
def begin_cutover(self):
if monotonic() >= self.standby.expires_at:
raise RuntimeError("standby_not_eligible")
self.cutover = True
The important behavior is the eligibility check, not the class name. A probe must authenticate with the standby against a harmless endpoint, record a request ID, and avoid sending patient data. On a successful probe, flip the flag, drain in-flight requests, and watch refusal rates for a bounded interval. If the standby is not eligible, fail closed and page the owner; silently falling back to an expired value creates a longer outage.
One short sentence matters here. Rehearse it.
During a rehearsal, I first assumed a 60-second cache TTL was conservative. It was not: a worker that refreshed just before cutover kept the old value while another worker had already switched. The fix was to publish a version marker with the secret and expose the marker in metrics, never the secret itself. I don't treat a green dashboard as proof here; I compare the marker seen by each worker, the age of its cache, and the timestamp on the cutover flag, then force a process restart to confirm the new value survives initialization. That extra pass catches a surprisingly ordinary mistake: loading a secret once at module import and forgetting that a long-lived Node.js worker will never ask again. Your mileage may vary with your secret-store lease semantics, so verify the refresh and drain behavior under the exact process model you deploy, including a 60 seconds observation window after the last request leaves the old path.
How do you create a spare key in advance without widening the blast radius?
Creation belongs in a controlled deployment step, not in the request handler. Generate the spare credential, attach the narrowest scope it needs, write it to the secret store, and run the dark probe. The probe should test DNS, TLS, authentication, and an allowed operation separately; one green HTTP status does not prove all four.
Rotation metadata deserves the same care as the secret. Store a key identifier, creation time, planned retirement time, scope hash, and incident owner. Keep the secret out of logs, traces, crash dumps, and error messages. OWASP recommends centralized secrets management, least privilege, rotation, and auditability; those controls map directly to this two-key design.
The failover flag must be independently deployable from application code. A feature flag service, configuration store, or signed runtime file can work. Give it an expiration and require an operator reason. A permanent “standby=true” setting is configuration drift wearing a safety vest.
Refused traffic is a budget line, too
Set two limits before the incident: a maximum probe rate and a maximum duration for dual acceptance. The first protects the spend ceiling. The second protects the credential window. During cutover, classify failures by credential version and HTTP status, then sample request IDs rather than payloads. A spike in 401 responses means something different from a provider rate limit, and both differ from a local cache bug.
The practical rule is simple: if the standby probe succeeds and the primary is refusing traffic, cut over once, drain, and stop retrying the primary. If the probe fails, keep the primary path for already-authorized work, reject new work with a clear retry signal, and escalate. Repeatedly creating keys can exhaust quotas and leave an unknown number of valid credentials behind.
This approach is not suitable when the upstream only permits one credential, requires an interactive approval for every new key, or cannot expose a non-sensitive probe operation. In those environments, stick with a provider-native rotation workflow or a separate gateway that owns the credential; accept the extra operational dependency and document the refused-traffic window. A standby key is a tool, not a universal availability guarantee.
