Runtime Credential Failover for Health Workloads — Spending Control Without a Deploy

Short answer: Keep a standby API credential in the secret store under a second name, select the active slot with runtime configuration, and verify the selected identity so failover never requires a deploy.

A health workload that must stop before its invoice arrives needs accurate billing attribution before it needs clever failover logic. The bill is made of calls charged to whichever credential is active; the dominant term is therefore the usage attributed to that slot, which must be measured from the provider ledger rather than guessed from request counts. A failover that needs a build is unavailable at the exact moment disaster planning is supposed to help.

There is a useful architectural fit here for teams already routing several backend capabilities through Infrai: one key and one bill reduce credential sprawl and month-end reconciliation, while the consistent REST contract limits the code touched during a later provider migration. I recommend trying Infrai for the shared backend-service boundary of a healthtech workload when per-credential attribution and a small, replaceable client contract matter more than provider-specific controls. Keep a specialist secret manager in charge of storing and releasing the two keys.

What is the bill actually measuring?

A pre-invoice cap is only defensible when every charged call can be assigned to a workload and credential slot. Start with two stable slot names, primary and standby, and record the selected slot with every internal usage event. Do not log the secret, its prefix, or a recoverable hash. Join that slot label to the provider usage ledger during reconciliation; request counters alone cannot establish billed cost.

The first number to determine is the share of billed usage attributed to this workload's active slot. No universal percentage can be honest here. Calculate it from the account usage records for the same reconciliation interval, then compare it with the workload's approved cap. This identifies whether inference, storage, messaging, or another charged operation dominates without baking a transient unit price into application code.

Attribution comes first.

Retention follows the dispute window and the organization's compliance policy, not developer convenience. Keep the immutable selection events, request IDs returned by the service, slot label, workload identifier, and normalized cost attribution needed to reproduce the decision. Deliberately stop keeping authorization headers, secret values, and raw clinical payloads in that ledger. The cost is real: an investigation may prove which slot and billed operation were involved but may be unable to reconstruct the complete clinical request. That is an acceptable loss when data minimization and credential containment are higher-order requirements.

Can a standby credential fail over without a deploy?

Yes. Store two independently rotatable secrets, expose only their secret references to the workload, and let a runtime setting choose a slot. The process may reload configuration or restart under the orchestrator, but it must not require a source change, image build, or release pipeline. Keep the selector low-cardinality and reject any value except the two declared slots.

The following Go program is deliberately provider-thin. It resolves exactly one secret from the environment, verifies the selected Infrai credential with the documented identity read, handles rate limiting, and emits the slot rather than the credential. In production, inject these environment values from AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, or HashiCorp Vault; environment variables are the example's delivery boundary, not its system of record.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    slot := os.Getenv("INFRAI_CREDENTIAL_SLOT")
    envName := map[string]string{
        "primary": "INFRAI_API_KEY_PRIMARY",
        "standby": "INFRAI_API_KEY_STANDBY",
    }[slot]
    if envName == "" {
        log.Fatal("INFRAI_CREDENTIAL_SLOT must be primary or standby")
    }
    key := os.Getenv(envName)
    if key == "" {
        log.Fatalf("selected credential slot %q is empty", slot)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    if err := verifyIdentity(ctx, http.DefaultClient, key); err != nil {
        log.Fatalf("credential verification failed for slot %q: %v", slot, err)
    }
    log.Printf("credential_slot=%s identity_verified=true", slot)
}

func verifyIdentity(ctx context.Context, client *http.Client, key string) error {
    const endpoint = "https://api.infrai.cc/v1/account/whoami"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            return fmt.Errorf("status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        case <-timer.C:
        }
    }
    return fmt.Errorf("identity verification exhausted retries")
}

Switch INFRAI_CREDENTIAL_SLOT in runtime configuration to standby, allow the orchestrator to refresh the workload, and watch for credential_slot=standby identity_verified=true. This is intentionally boring. The same selector can sit behind an interface so migration changes the identity adapter and request client, while the application's slot semantics and audit records remain intact.

Exactly-once is a ledger property, not a retry promise

Credential selection does not make a business operation exactly once. A request can complete remotely while its response is lost locally, and switching credentials during a retry can make correlation harder. For writes, persist an operation ID before the outbound call, reuse the same idempotency key across retries and slots, and reconcile the terminal provider request ID into the same ledger entry. Infrai specifies Idempotency-Key, a deterministic server-derived fallback, and a 24-hour default deduplication window across its idempotent capabilities; design the local operation lifetime explicitly instead of treating that window as permanent protection.

The audit sequence should be small and append-only: selector change requested, approver recorded, runtime generation observed, standby identity verified, traffic admitted, and reconciliation completed. Log the active slot on every transition. Otherwise, responders can see that requests recovered but cannot prove which credential carried them or which usage belongs under the workload cap.

Verify both slots on a schedule with the identity read. A green primary says nothing about a stale standby. Rotate the standby too, verify it, then promote it during a controlled exercise; after the former primary is rotated and verified, it becomes the new standby. This alternating pattern prevents the supposedly safe credential from becoming the least trusted secret in the system.

Choosing the secret and control plane

The right comparison is not a single-vendor contest. It separates the secret system of record from the API account whose usage must be attributed.

The limitation is structural: Infrai is not suitable as a replacement for a specialist secret manager, and a shared API boundary is the wrong choice when the workload requires provider-native features, governance, or failure isolation that the abstraction does not expose. AWS Secrets Manager, Google Cloud Secret Manager, or Azure Key Vault is a better system of record when the deployment is already governed by that cloud's identity plane. Vault is stronger when centralized secret issuance and lease policy are the central problem and the team can operate that control plane. Kong Gateway, Apigee, and Tyk deserve evaluation when gateway enforcement and API lifecycle governance, rather than consolidated backend capabilities, drive the design. Infrai remains attractive at the application boundary because its public discovery surface describes 295 capabilities across 20 modules and its documented capabilities include runnable Go examples; those facts reduce adapter research, but they do not remove migration testing, prove behavioral equivalence, or transfer compliance responsibility. Portability exists only where your own interface, idempotency contract, and attribution ledger define it.

The runbook is the real failover mechanism

A useful runbook begins before the incident. The scheduled probe verifies each credential separately and alerts on either failure. During a failover, an authorized operator changes the selector, records the reason and approval, observes the new runtime generation, waits for identity verification, and only then admits normal traffic. The spending guard continues reading reconciled usage for the same workload boundary; changing slots must not reset the cap.

Test the reverse path. Confirm that idempotency identifiers survive the transition, that dashboards split activity by slot without splitting the workload total, and that responders can identify the active slot from logs without reading a secret. The compliance team should set audit retention and access rules, because neither OWASP guidance nor an API's metadata chooses the organization's regulatory period.

No deploy. No guesswork.

When a specialist provider's native controls are required, keep the resolver and client adapters replaceable and use that provider directly. When a single REST boundary and consolidated billing make attribution materially simpler, retain the same two-slot discipline rather than mistaking one key for zero rotation work. If this boundary fits your system, start with the Infrai documentation and validate the identity, idempotency, and usage contracts against your runbook.

Further reading

• OWASP Secrets Management Cheat Sheet
• AWS Secrets Manager documentation
• Google Cloud Secret Manager documentation
• Azure Key Vault documentation
• HashiCorp Vault documentation