Short answer: Select a standby credential at request time, but switch only after an operator or a narrowly defined authentication signal confirms the primary cannot be used. For a logistics prepaid-balance monitor, the hard part is preserving which account pays for each balance read and each alert, not merely keeping the API call alive. A successful request against the wrong billing account is a failure.
How can a standby API credential fail over without a deploy?
Consider a bounded incident exercise: I would put one primary and one standby credential in a secrets store before the exercise, each bound to an explicitly recorded billing account. At 09:00, the primary becomes unusable while a scheduled check is due; the process is already running and a deployment is off the table. This is a drill, not a claim about an observed outage. The invariant is stricter than availability: no check may silently change the account whose balance it reports, and no alert may omit the account and credential version used to obtain the reading.
This matters because a standby credential for another account may return a perfectly plausible balance. The monitor might then decide that a prepaid account has enough funds while the intended account is running low. Fail closed if the standby's billing-account binding differs from the intended account.
The wrong balance still looks valid.
That is the trap. An authentication success tells us nothing about whether the monitored logistics account owns the credential; account identity must be checked before the result is trusted, including after a switch back to the primary. With two replicas polling the same account, their selector reads also need a consistent view or an explicit transition window: otherwise a legitimate update can yield mixed versions across checks and make incident review much harder than the selector's apparent simplicity suggests.
A deployment-time environment variable is easy to audit, but rotating it requires restarting the process and coupling incident response to release machinery. A mutable runtime selector avoids a deploy, yet broad write access turns an operational control into a billing-integrity risk. A broker can centralize policy, but adds a dependency to the balance-check path and a new on-call surface.
| Approach | Attribution control | On-call and capacity consideration |
|---|---|---|
| Restart with a new secret | Account binding reviewed in release configuration | Recovery waits on restart and rollout capacity |
| Runtime selector with versioned secrets | Enforce account equality on every selection | Limit selector writers; keep a tested rollback path |
| Credential broker | Enforce account binding at one boundary | Budget for broker availability and its request load |
I would size any path for the peak scheduled checks plus bounded retry traffic, not the average alert rate. A recovery-time objective should include detection, authorized switch, the next successful same-account read, and alert delivery. An SLO for completed checks should exclude readings attributed to the wrong account, even if the upstream returned success.
No account match, no result.
How does the request path reject a mismatched standby?
The following Go example uses an abstract secrets reader so the selector can change without rebuilding the service. The selector holds only a version name; secret material remains in the store. The caller supplies the expected account from its own job configuration and records the returned version alongside the reading.
package balance
import (
"context"
"errors"
)
type Credential struct {
Token string
AccountID string
Version string
}
type Store interface {
ActiveVersion(context.Context) (string, error)
ReadCredential(context.Context, string) (Credential, error)
}
func Select(ctx context.Context, store Store, expectedAccount string) (Credential, error) {
version, err := store.ActiveVersion(ctx)
if err != nil {
return Credential{}, err
}
credential, err := store.ReadCredential(ctx, version)
if err != nil {
return Credential{}, err
}
if credential.Token == "" || credential.Version != version ||
credential.AccountID == "" || credential.AccountID != expectedAccount {
return Credential{}, errors.New("credential attribution check failed")
}
return credential, nil
}
The check is intentionally per operation: caching the selected version indefinitely would turn the runtime switch into a restart-dependent feature. In a real request path, use a bounded context deadline, never log the token, and attach the expected account and selected version to the check's audit record without putting secrets in telemetry. If the selector changes between selection and retrieval, the returned version check rejects an inconsistent result; the store still needs a consistent version-to-secret contract.
What should the drill prove before a real incident?
Test the two credentials against the same expected billing account, then revoke the primary in a controlled environment and switch the selector with a separately authorized operation. Verify that an in-flight check completes under its original version or fails with a clear status; do not blindly retry a potentially billable operation merely because authentication failed. For a read-only balance check, confirm the upstream's retry semantics before enabling one bounded retry with the standby. Inject an unreachable secrets store and a standby bound to another account. Both must produce an unavailable check, never a healthy balance signal.
Observe the age of the last verified balance, failed selection counts by reason, check latency, and the active credential version. Alert on stale verification rather than treating a successful selector update as recovery. Keep credentials out of metric labels; version identifiers and account identifiers need controlled cardinality and access as well. OWASP's secrets guidance supports central inventory, rotation, least-privilege access, and auditing, but it does not make a credential swap an authorization decision on your behalf.
Limitations matter here: this pattern is not suitable when the underlying billing account is exhausted, both credentials share the same revoked permission, or the upstream cannot establish which account a credential belongs to. In those cases, stop claiming the prepaid balance is known and escalate the account or provider failure. The useful recovery criterion is a fresh, same-account reading followed by the expected alert decision.
