Staleness Budgets: Designing Freshness Guarantees for Supplier Price and Stock Feeds

Every integration that reads another company's inventory is a claim about the past dressed up as a claim about the present. You cached a number at 06:04. A customer reads it at 14:52. Somewhere in between, a warehouse in Edmonton shipped the last four casings of that size to somebody else, and nobody told you.

The usual reaction is to shorten the refresh interval and move on. That reaction is wrong in an interesting way: it treats freshness as a knob rather than as a property you can specify, measure, and hold yourself accountable to. What follows is a design for treating it as the latter — a staleness budget per field class, derived from measured change rates and from the cost of being wrong at a specific decision point.

Concrete setting, because abstractions about "entities" get slippery fast: a tire retailer pulls sizes, brands, availability and landed cost from several distributor feeds. Someone asks whether a 225/45R17 can go on the vehicle Thursday. Answering requires three facts that belong to other companies and one that belongs to you. Everything below is a design exercise. All numbers in code, tables and worked examples are illustrative and synthetic — placeholders chosen to make the arithmetic legible, never real supplier data, real cost, or real availability.

The only honest question about somebody else's inventory

"Is it available?" is not answerable. It is a question about the current state of a system you cannot observe, and the observation you can make takes 200 milliseconds to travel and arrives describing a moment that has already passed.

The answerable version is: given that I last observed this value N seconds ago, what is the probability it has since changed, and what does being wrong cost me here? Both halves matter. A landed cost that drifted a little on a browse page costs mild embarrassment. The same drift on a confirmed reservation costs either margin or a phone conversation nobody enjoys. Same data, same age, completely different tolerance.

So the unit of design is not the cache. It is the pair (field class, decision). A staleness budget is the maximum age at which a value may still be used for a particular decision, chosen so the probability of being wrong stays under a target you picked on purpose. Write that budget down. Enforce it in code. Alarm when you violate it. Everything else in this article is machinery for making that sentence true.

Model freshness as data, not as a boolean

The first structural mistake is storing available: true. The second is storing available: true, updated_at: <timestamp> and treating those two columns as unrelated. They are one value. The timestamp is not metadata; it is half the meaning.

Three distinct instants deserve separate columns, and collapsing them will cost you a debugging afternoon eventually:

• observed_at — when your process received the bytes. This is about your pipeline. It is the only one of the three you can fully trust, because you generated it from your own clock.
• source_as_of — the instant the supplier asserts the value was true. Present in good feeds, absent in most. Untrusted input: clamp it, never let it exceed observed_at by more than a small skew allowance, and record when you had to clamp it.
• valid_until — a supplier-asserted expiry, when they publish one. Rare, and worth honouring when it appears, because it is the only case where somebody else has told you their own budget.

Why not one column? Because retries, queue backlogs and replays decouple them. A delta message produced at 06:04 that sat in a stalled consumer until 09:30 has source_as_of = 06:04 and observed_at = 09:30. If you kept only one, you would either believe the data is three hours fresher than it is, or you would discard a perfectly good historical fact. You need both to compute age and to detect pipeline lag, which are different alarms with different owners.

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Generic, TypeVar

T = TypeVar("T")


class FieldClass(str, Enum):
    PRICE = "price"
    AVAILABILITY = "availability"
    LEAD_TIME = "lead_time"
    DESCRIPTIVE = "descriptive"


@dataclass(frozen=True)
class Observed(Generic[T]):
    """A value plus everything needed to reason about how much to trust it."""

    value: T
    field_class: FieldClass
    source: str
    observed_at: datetime
    source_as_of: datetime | None = None
    valid_until: datetime | None = None

    @property
    def effective_as_of(self) -> datetime:
        """Prefer the supplier's assertion, but never let it run ahead of us."""
        if self.source_as_of is None:
            return self.observed_at
        return min(self.source_as_of, self.observed_at)

    def age(self, now: datetime) -> timedelta:
        return now - self.effective_as_of

    def expired(self, now: datetime) -> bool:
        return self.valid_until is not None and now > self.valid_until


def utcnow() -> datetime:
    return datetime.now(timezone.utc)

On the wire, make "we do not know" representable. A boolean plus an optional timestamp lets an unknown masquerade as a fresh false, and a discriminated union does not:

export type FieldClass = "price" | "availability" | "leadTime" | "descriptive";

export interface Observed<T> {
  readonly value: T;
  readonly fieldClass: FieldClass;
  readonly source: string;
  readonly observedAt: string;    // RFC 3339, always UTC, generated by us
  readonly sourceAsOf?: string;   // supplier's own assertion, when published
  readonly validUntil?: string;   // supplier-asserted expiry, when published
}

export type Freshness =
  | { readonly state: "fresh"; readonly ageSeconds: number }
  | { readonly state: "stale"; readonly ageSeconds: number; readonly budgetSeconds: number }
  | { readonly state: "unknown"; readonly reason: "never-observed" | "upstream-down" };

export interface Answer<T> {
  readonly data: Observed<T> | null;
  readonly freshness: Freshness;
}

That unknown variant is the whole point. A downstream renderer that has to destructure it cannot accidentally paint a confident green badge over a supplier outage.

Field classes decay at very different rates

Lump every attribute of a product into one record and you have implicitly declared that they all rot at the same speed. They do not, and the spread is enormous.

A tire's section width, aspect ratio and rim diameter never change for a given part number — those are identity, not state. The markings moulded into the sidewall are fixed at manufacture. A load index is a property of the design. If you are polling a supplier every fifteen minutes to re-learn that a tire is still 225 millimetres wide, you are spending your rate limit on the least interesting bytes in the payload.

Availability, by contrast, is a counter that other people decrement all day. Landed cost changes when a price file lands, when freight terms shift, or when a distributor's own pricing cycle turns over — bursty, not smooth. Lead time is a derived estimate that moves with warehouse routing.

Those budgets are placeholders for the shape of the argument, not measured production values. The point is the four orders of magnitude between the top row and the bottom one. Any design that cannot express that spread is throwing away information you already have.

Field classes also let you split storage sensibly. Descriptive attributes belong in a slowly-changing catalogue table refreshed by the nightly snapshot. Availability belongs in a hot table with a narrow row and an index built for age queries. Mixing them means every availability tick rewrites a row containing a 400-character marketing description, which is a fine way to melt your write amplification.

One global TTL is the wrong abstraction

A single TTL forces one number to satisfy every consumer, so it converges on whichever consumer is loudest. Set it to fifteen minutes and 95% of your request budget goes to re-fetching data that changes twice a year. Set it to six hours and you will confidently offer a size that went to zero before lunch.

There is a subtler problem. A TTL is a property of a record, but staleness tolerance is a property of an edge — the specific read, by a specific consumer, for a specific purpose. The same availability row is simultaneously fresh enough for a category listing page and far too old for a confirmation screen. Attaching the number to the row erases that distinction and forces the strictest consumer's requirement onto everyone.

The third problem is that TTLs are uniform in time while demand is not. In Calgary a chinook can move the temperature twenty degrees in an afternoon, and the volume of people suddenly interested in changeover work follows the thermometer with a lag of about a day. During the October ramp, availability on popular winter sizes churns hard; in July, the same rows sit untouched for weeks. A fixed interval is either wasteful in summer or dangerously slack in autumn, and it is usually both, at different times, for different SKUs.

So: budgets attach to (field class × decision), and refresh scheduling is a separate concern driven by observed volatility and observed demand. Conflating policy with scheduling is the root error that a global TTL encodes.

Measuring decay from your own history

You cannot derive a budget from first principles. You derive it from what your own data did last month.

Keep an append-only change log: every time a poll produces a value different from the stored one, write a row with the supplier, the SKU, the field class, the previous value, the new value, and both timestamps. This log is cheap — it only grows on change — and it is the input to everything else.

The naive estimator is changes_detected / total_observation_hours. It is biased low, and knowing why matters. If you poll every 20 minutes and a value bounces from 4 to 0 to 4 between polls, you record zero changes. Polling gives you interval-censored data: you learn only whether the value moved during a window, never how many times.

Under a constant-hazard assumption, the correction is one line of algebra. If changes arrive as a Poisson process with rate λ, the probability that at least one change occurs in a window of length T is 1 - exp(-λT). Estimate that probability as the fraction of polls that detected a difference, then invert:

import math


def hazard_from_polls(polls: int, changes_detected: int, interval_seconds: float) -> float:
    """Interval-censored estimate of change rate, in changes per second.

    We observe only whether a value differed between two consecutive reads, so a
    raw change count understates reality. Inverting the Poisson survival function
    recovers the underlying rate from the detection fraction.
    """
    if polls <= 0 or changes_detected <= 0 or interval_seconds <= 0:
        return 0.0
    detection_fraction = min(changes_detected / polls, 0.999)
    return -math.log(1.0 - detection_fraction) / interval_seconds


def ttl_for_error_budget(rate_per_second: float, target_error: float) -> float:
    """Oldest age at which P(value already moved) stays under target_error."""
    if rate_per_second <= 0.0:
        return math.inf
    if not 0.0 < target_error < 1.0:
        raise ValueError("target_error must be strictly between 0 and 1")
    return -math.log(1.0 - target_error) / rate_per_second

Bucket the estimate. Per-SKU rates over-fit wildly on long-tail parts you have observed nine times; a global rate hides the fact that a common winter size behaves nothing like a niche commercial fitment. Something like (supplier × field class × demand decile) is usually the right granularity, with shrinkage toward the bucket mean for sparse cells.

And then the honest caveat: constant hazard is a lie you adopt deliberately. Price changes are not memoryless — they cluster at whatever hour the distributor's batch job runs. Availability hazard tracks order flow, which has a weekday shape and a weather shape. Treat the exponential model as a floor for reasoning, then override it with the two things you actually know: known publication times, and known demand spikes.

From hazard rate to a number you can defend

With a rate in hand, the budget falls out of a target error probability you choose per decision.

Survival under constant hazard is P(unchanged after age t) = exp(-λt), so the age at which you first exceed an error tolerance ε is t = -ln(1 - ε) / λ. Small ε makes the numerator approximately ε itself, which gives a useful mental shortcut: the budget is roughly your error tolerance divided by the change rate.

Illustrative arithmetic, synthetic figures throughout. Suppose a popular winter size at one distributor shows a detected-change fraction of 0.08 per 20-minute poll during the autumn ramp. Then λ ≈ -ln(0.92) / 1200 s ≈ 6.95 × 10⁻⁵ per second. For a browse page where you accept a 10% chance of being wrong, the budget is -ln(0.90) / 6.95e-5 ≈ 1,516 seconds, roughly 25 minutes. For a firm commitment where you accept 1%, it collapses to -ln(0.99) / 6.95e-5 ≈ 145 seconds. Same data, same rate, an order of magnitude apart, and the second number is short enough that the only sane implementation is to revalidate at the moment of commitment rather than to poll harder.

Two refinements that earn their complexity:

Asymmetry. Not all changes hurt equally. Availability falling to zero is expensive; availability rising from 4 to 9 is free. Landed cost rising costs you margin; falling costs you nothing you will notice today. Estimate one-sided rates — the hazard of a harmful transition — and set budgets against those. In practice this roughly doubles usable budgets, because harmful moves are typically a minority of moves.

Decision-weighted ε. Publish a small table of tolerances rather than scattering magic numbers: listing pages 10%, quote generation 3%, reservation confirmation 1%, fleet commitments 0.5%. Whether those are the right values is a business argument, and the table is where you have it — once, in the open, instead of implicitly in seventeen different cache decorators.

A policy evaluator you can unit-test

Budgets are worthless as documentation. They need to be a function that returns a verdict, so that the decision to serve stale data is a recorded event rather than an accident.

BUDGET_SECONDS: dict[FieldClass, float] = {
    FieldClass.PRICE: 6 * 3600,
    FieldClass.AVAILABILITY: 20 * 60,
    FieldClass.LEAD_TIME: 24 * 3600,
    FieldClass.DESCRIPTIVE: 30 * 24 * 3600,
}

DEMAND_TIER_MULTIPLIER: dict[str, float] = {"hot": 0.25, "warm": 1.0, "cold": 4.0}


class Verdict(str, Enum):
    SERVE = "serve"
    SERVE_LABELLED = "serve_labelled"
    REVALIDATE_FIRST = "revalidate_first"
    REFUSE = "refuse"


@dataclass(frozen=True)
class Decision:
    verdict: Verdict
    age_seconds: float
    budget_seconds: float

    @property
    def budget_violated(self) -> bool:
        return self.age_seconds > self.budget_seconds


def budget_for(field_class: FieldClass, demand_tier: str) -> float:
    return BUDGET_SECONDS[field_class] * DEMAND_TIER_MULTIPLIER[demand_tier]


def evaluate(
    obs: Observed[object] | None,
    *,
    now: datetime,
    field_class: FieldClass,
    demand_tier: str = "warm",
    binding: bool,
) -> Decision:
    """Decide what may be done with a cached value.

    `binding` marks decisions that create an obligation to the customer. Those
    never accept stale input; everything else may degrade gracefully.
    """
    budget = budget_for(field_class, demand_tier)
    if obs is None:
        return Decision(Verdict.REVALIDATE_FIRST if binding else Verdict.REFUSE, math.inf, budget)

    age = obs.age(now).total_seconds()
    if obs.expired(now):
        return Decision(Verdict.REVALIDATE_FIRST, age, budget)
    if age <= budget:
        return Decision(Verdict.SERVE, age, budget)
    if binding:
        return Decision(Verdict.REVALIDATE_FIRST, age, budget)
    if age <= budget * 4.0:
        return Decision(Verdict.SERVE_LABELLED, age, budget)
    return Decision(Verdict.REFUSE, age, budget)

Four verdicts, not two. The interesting one is SERVE_LABELLED: past budget, not yet absurd, safe to show provided the interface admits its age. The budget * 4.0 cliff is a policy choice — beyond some multiple, a number stops being approximate and starts being fiction.

Pull, push, and the hybrid that survives contact

Every real integration ends up with three mechanisms, and each one exists because the other two have a specific hole.

The nightly full snapshot is your ground truth reset. It is the only mechanism that catches deletions, because deltas tell you what changed and are silent about what vanished. It bounds worst-case drift at 24 hours regardless of what else broke. Run it against a staging table, compare row counts against the previous run, and refuse to promote a snapshot whose count moved more than a threshold — a truncated file that parses cleanly is the classic way to zero out a catalogue at 03:00.

Intraday deltas are how you get minutes instead of hours. Webhooks, a message queue, an incremental endpoint keyed on a cursor. The rule with deltas is simple and universally violated: never assume completeness. Demand a monotonic sequence number per stream, track the highest one you have processed, and alarm on gaps. If the supplier will not give you a sequence, you cannot detect loss, and you must treat the delta stream as a latency optimisation rather than a correctness mechanism.

On-demand revalidation at the decision point is the only thing that makes commitments defensible. When someone is about to be told "yes, Thursday works," you spend one synchronous request against the supplier and you spend it right then. The budget arithmetic above showed why: a 1% tolerance on a volatile field yields a budget measured in low minutes, and no polling schedule you can afford will keep every SKU that fresh. You do not need every SKU that fresh. You need the one that is about to become a promise.

Hybrid, therefore, is not indecision. It is three different guarantees composed: bounded worst case from the snapshot, good typical case from deltas, and exactness where it counts from revalidation.

Conditional requests make aggressive polling cheap

Here is the mechanism people skip, and it is the one that changes the economics.

An HTTP 304 Not Modified costs a round trip and a few hundred bytes, and it tells you something enormously valuable: the value you already hold is still current as of now. That resets observed_at without transferring, parsing, or writing anything. Freshness without payload. Once you internalise that, polling frequency stops being a bandwidth question and becomes a request-count question, and request counts are far cheaper to buy.

from dataclasses import replace

import httpx


@dataclass(frozen=True)
class CacheEntry:
    payload: dict | None
    etag: str | None
    last_modified: str | None
    observed_at: datetime


def revalidate(client: httpx.Client, url: str, entry: CacheEntry) -> CacheEntry:
    """Refresh an entry, transferring bytes only when the resource moved."""
    headers: dict[str, str] = {}
    if entry.etag:
        headers["If-None-Match"] = entry.etag
    if entry.last_modified:
        headers["If-Modified-Since"] = entry.last_modified

    response = client.get(url, headers=headers, timeout=5.0)
    if response.status_code == 304:
        return replace(entry, observed_at=utcnow())

    response.raise_for_status()
    return CacheEntry(
        payload=response.json(),
        etag=response.headers.get("ETag"),
        last_modified=response.headers.get("Last-Modified"),
        observed_at=utcnow(),
    )

Details that bite:

Last-Modified has one-second granularity, so two changes inside the same second are indistinguishable and you can pin a stale body indefinitely. Prefer ETag when both are offered. Weak validators (W/"abc") mean semantic equivalence, not byte equality — fine for freshness, useless if you are checksumming.

Many distributor APIs support neither. Two fallbacks: ask for a lightweight digest endpoint returning a version stamp per SKU group, or hash the normalised payload yourself. Self-hashing does not save bandwidth, but it saves parsing, writes, downstream invalidation, and — importantly — it prevents a re-serialised-but-identical payload from creating a spurious "change" that corrupts your rate estimates. Normalise before hashing: sort keys, drop server-generated request IDs, drop the response timestamp.

Scheduling: priority, jitter, and a shared bucket

Fixed-interval polling spends uniform effort on non-uniform value. Replace it with a priority score and a work queue.

A serviceable score multiplies three factors: how much demand the SKU sees, how volatile its field class is at that supplier, and how far through its budget the current value has aged. Something like demand_weight × λ̂ × (age / budget), recomputed lazily when an item is popped. Items whose age exceeds budget dominate naturally, items nobody looks at sink, and a SKU that becomes popular climbs within one cycle.

Demand weight is where local knowledge pays. Sizes that fit the vehicles common on Deerfoot in December deserve tighter refresh than a fitment you have quoted twice this year, and the popularity of a given winter fitment versus an all-weather option shifts with the forecast, not with the calendar. Feed the score from actual read counts over a trailing window and it tracks reality without anyone maintaining a list.

Then jitter, because synchronised timers are a self-inflicted denial of service. Anything scheduled with interval and no spread will converge into a convoy — a nightly job at midnight, 40,000 SKUs whose timers were all seeded at deploy, a supplier that starts returning 429s at exactly 00:00:03.

import random
import time


def next_poll_at(now: float, interval_seconds: float, *, spread: float = 0.15) -> float:
    """Fixed cadence with proportional jitter so thousands of items never align."""
    return now + interval_seconds * random.uniform(1.0 - spread, 1.0 + spread)


def backoff_delay(attempt: int, *, base: float = 0.5, cap: float = 60.0) -> float:
    """Full-jitter exponential backoff: growing ceiling, uniform draw beneath it."""
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0.0, ceiling)


class TokenBucket:
    """Per-supplier rate limit shared by every worker.

    This in-process version is the reference model; production state lives in a
    single store so N workers draw from one pool rather than N pools.
    """

    def __init__(self, rate_per_second: float, burst: int) -> None:
        self.rate = rate_per_second
        self.burst = burst
        self._tokens = float(burst)
        self._checked = time.monotonic()

    def acquire(self, cost: float = 1.0) -> float:
        """Seconds to wait before proceeding; 0.0 means go immediately."""
        now = time.monotonic()
        self._tokens = min(self.burst, self._tokens + (now - self._checked) * self.rate)
        self._checked = now
        if self._tokens >= cost:
            self._tokens -= cost
            return 0.0
        return (cost - self._tokens) / self.rate

The shared-bucket point deserves emphasis. Per-process limits are not limits. Six workers each politely capped at two requests per second produce twelve, and when autoscaling adds four more during the autumn ramp you are at twenty and the supplier's abuse detection has opinions. One bucket per supplier, in one place, with cost weighted so a full snapshot pull consumes more tokens than a conditional probe.

Failing well: breakers, backoff, and negative caching

Retry policy for feeds is mercifully simple because reads are idempotent. Retry GET freely; budget the retries per logical request rather than per attempt, so a request with a 3-second deadline does not spend 40 seconds heroically retrying into a supplier that is plainly down.

Circuit breakers matter more here than in most integrations, because the failure mode is not just latency — it is that a hung supplier holds workers hostage while every SKU in the queue ages past its budget. Trip on error rate over a rolling window rather than on a raw count. Half-open with a single probe, and choose the probe deliberately: a known-stable descriptive endpoint, not the heaviest availability query. When the breaker is open, every read for that supplier returns unknown immediately, which is a far better answer than a 30-second timeout.

Negative caching is the underrated half. A SKU that returns 404 will keep returning 404, and re-asking 3,000 times an hour helps nobody. Cache the negative — with its own, shorter budget, because "does not exist" reverses more often than you would think when a distributor onboards a line. The hard rule: never negative-cache a 5xx or a timeout. Those mean "I could not ask," not "the answer is no," and conflating them turns a ninety-second blip into an hour of phantom unavailability. That distinction is worth a separate cache state, not a shared one.

Serve stale with a label, or fail closed

This is the decision that people get wrong most often, and the reason is that they ask "how stale is it?" when the question is "what happens if it is wrong?"

Two consequence classes, and you should be able to name which one every read belongs to:

Wasted click. A listing shows a size as available; the customer clicks through; the detail view revalidates and corrects. Cost: mild annoyance. Serve stale, label it, move on. Refusing to render anything because a feed is 40 minutes old produces a blank page, which is strictly worse than an approximate one.

Wrong commitment. Someone is told a specific set can go on Thursday. That sentence creates an obligation. If the casings are not there, the cost is a rescheduled reservation, a wasted bay hour, and a driver who took an afternoon off work. Fail closed: revalidate synchronously, and if the supplier will not answer, say "let me confirm and get back to you" rather than guessing.

The boundary between those worlds should be a type, not a convention. Make the commitment path take a Confirmed<T> that can only be constructed from a fresh revalidation, and the compiler will enforce what a code review will eventually miss. In practice this means the reservation flow — the one behind the scheduling page — physically cannot accept a value that came out of the cache without a successful probe attached.

Same logic applies to work that travels. A van going out to a customer's site carries a fixed load of inventory, and being wrong about what is on it is not a wasted click but a wasted trip across the city. Anything dispatched to a location in our coverage map gets the fail-closed treatment.

Out-of-order updates and idempotent upserts

Distributed delivery reorders things. A retried webhook from 06:04 arrives after a fresh one from 06:09 and, if your upsert is unconditional, quietly resurrects a five-minute-old price. This bug is invisible in testing and produces exactly the kind of intermittent wrongness that costs a week to trace.

The fix is a monotonic version per source and a conditional write.

create table supplier_offer (
  supplier_id      text        not null,
  sku              text        not null,
  price_cents      integer     not null,
  currency         char(3)     not null default 'CAD',
  available_units  integer,
  lead_time_days   smallint,
  source_version   bigint      not null,
  source_as_of     timestamptz,
  observed_at      timestamptz not null,
  valid_until      timestamptz,
  tombstoned       boolean     not null default false,
  primary key (supplier_id, sku)
);

create index supplier_offer_age_idx
  on supplier_offer (supplier_id, observed_at desc)
  where tombstoned = false;

And the write, which refuses to move backwards:

insert into supplier_offer as existing (
  supplier_id, sku, price_cents, available_units,
  lead_time_days, source_version, source_as_of, observed_at
)
values ($1, $2, $3, $4, $5, $6, $7, now())
on conflict (supplier_id, sku) do update
   set price_cents     = excluded.price_cents,
       available_units = excluded.available_units,
       lead_time_days  = excluded.lead_time_days,
       source_version  = excluded.source_version,
       source_as_of    = excluded.source_as_of,
       observed_at     = excluded.observed_at
 where excluded.source_version > existing.source_version;

The where on the do update branch is the entire mechanism. Replays become no-ops, out-of-order arrivals are dropped, and the statement is idempotent under any delivery order. Count the dropped ones — a rising rate of rejected-as-older writes is a good early signal that a consumer is lagging.

When the supplier publishes no version, synthesise one. Ranked preference: their own sequence number; failing that, source_as_of converted to microseconds; failing that, a per-batch counter combined with the batch's start time. Falling back to observed_at is legal but weak, because it makes the retry itself look newer than the original — precisely the case you were defending against.

Multi-source is where it gets genuinely awkward. If two distributors both stock a SKU, there is no single ordering, and a version vector keyed by source is the honest structure: keep each source's latest observation separately, and resolve at read time with an explicit rule (cheapest currently-available, or preferred-partner-first, or fastest to depot). Merging them into one row destroys the ability to answer "which supplier said that, and when?" — which is the first question anyone asks when a number looks wrong.

Tombstones instead of deletes, for the same reason. A row that disappears from a snapshot has been discontinued, and "we knew about this until Tuesday" is a much more useful state than a missing row that looks identical to a SKU you never ingested.

The reconciliation loop and the drift metric

Everything above is inference. Reconciliation is the part where you actually check.

Sample records on a schedule, fetch them fresh and unconditionally, and compare against what you were serving. The comparison yields the single most valuable number in the system: drift rate, the fraction of sampled records whose cached value was wrong while inside its budget. If that number exceeds the ε you designed for, your budgets are too generous and the arithmetic is not opinion any more.

Three design notes that make the difference between a useful loop and a decorative one:

Stratify the sample. Uniform sampling across 40,000 SKUs spends almost all its effort on the long tail, where nothing changes and nobody looks. Weight by read volume so the sample resembles the traffic. Report drift per stratum, because a 0.4% aggregate hiding 6% on your top fifty sizes is worse than useless.

Separate drift from loss. A record can be wrong because the value moved and the budget was too long (a budget problem), or because a delta message was never delivered (a pipeline problem). Distinguish them by checking whether the correct value's source version was one you had ever received. Different alarms, different owners, and conflating them sends the wrong team looking.

Sample the writes, not just the reads. Reconcile a slice of records that your delta stream claims to have updated recently. If a delta said "4 units" and the truth is 9, you have a transformation bug, not a freshness bug, and no amount of polling faster will fix it.

Surfacing age to humans and to downstream systems

Internally, freshness is a typed field on the response body. Not a header — headers get stripped by proxies, ignored by clients, and lost the moment somebody wraps your response in an envelope. Put it in the payload where the type system can see it, and make it non-optional so nobody can forget to handle it.

For humans, the design goal is to convert an engineering fact into a calibration cue without inducing anxiety. "Checked 6 minutes ago" does that. A green dot does not — it communicates certainty you have not earned. Rounding rules worth stealing: under 60 seconds say "just now"; under an hour, whole minutes; under a day, whole hours; beyond that, the date. Never round up into a friendlier bucket, because the entire value of the label is that it is conservative.

Context changes the wording more than the number does. Someone comparing all-season against winter-specific options on a browse page does not need a timestamp on every tile; a single "availability checked within the last 20 minutes" line at the top of the results is enough calibration. Someone on a confirmation screen deserves the exact age of the specific fact they are relying on, because that is the moment where the number becomes a promise.

For fleet customers the requirement is different again. A fleet operations integration consuming your API needs machine-readable freshness so their own planner can decide whether to trust a figure, and it needs the age of the oldest input to a derived number, not the newest. Composite freshness is a minimum, never an average.

Observability: histograms, violations, and the silence alarm

Instrument age at read time. Age at write time is a measure of your poller's diligence; age at read time is a measure of what customers actually experienced, and the two diverge badly whenever your scheduling priorities are misaligned with demand.

Then the alarm nobody builds until after it has hurt them: feed silence. An unchanged feed is almost always a broken feed, not a stable one. If a supplier's availability stream normally produces four thousand changes a day and produced eleven yesterday, something upstream is stuck — a credential expired, a cursor got pinned, a filter clause started matching nothing. Alarm on the absence of change, using a floor derived from that stream's own trailing history rather than a hand-picked constant.

Related pathologies worth their own checks: a 304 ratio that reaches exactly 100% for a whole day (a validator that stopped changing rather than content that stopped moving); record counts landing on suspiciously round numbers like 10,000 (a page limit silently truncating); and a snapshot whose row count drops by more than a few percent (a partial file). Each of these has produced a quiet catalogue corruption somewhere, and each is a five-line check.

Here is a read-log query that produces the top half of that table:

select
    field_class,
    count(*)                                                  as reads,
    percentile_disc(0.50) within group (order by age_ms)      as p50_age_ms,
    percentile_disc(0.99) within group (order by age_ms)      as p99_age_ms,
    round(avg((age_ms > budget_ms)::int)::numeric, 4)         as violation_rate
from freshness_read_log
where read_at >= now() - interval '7 days'
group by field_class
order by violation_rate desc;

Sample the read log rather than writing every read — one in a hundred is plenty for percentiles at any real volume, and the write amplification of logging every cache hit will outweigh the insight.

A worked example, entirely synthetic

Numbers below are illustrative placeholders invented for the arithmetic. They are not KMJ figures, not supplier figures, and not measured production values.

Setting: three distributors, 40,000 SKU-supplier pairs, an API allowance of 2 requests per second per supplier, so about 518,000 requests per supplier per day.

Naive uniform polling at 20 minutes needs 40,000 × 72 = 2.88M requests per day against each supplier. Over budget by 5.5×. This is where most teams give up and stretch the interval to two hours, quietly accepting a drift rate they never measure.

Now apply the structure. Split by field class: descriptive attributes ride the nightly snapshot and cost nothing intraday. Only availability and price need intraday attention, and only the demand-weighted head of the catalogue needs it often.

Total intraday across all three suppliers is roughly 595,600 requests, about 199,000 per supplier — comfortably inside the 518,000 allowance, with headroom for the autumn ramp when hot-tier membership swells. Bytes transferred are dominated by the 304s, so actual bandwidth is a fraction of the naive plan even though request volume is only modestly lower.

Freshness outcome, using the illustrative λ from earlier: hot-tier availability carries a p99 read age near 5 minutes against a 5-minute budget (hot tier multiplier 0.25 × 20 min), warm tier sits near 45 minutes against 20, which sounds like a failure until you notice warm-tier reads are browse traffic where the tolerance is 10% and the labelled-stale path is fine. Commitments bypass the cache entirely. Three different guarantees, one system, and every one of them is written down and measured.

Failure modes worth designing against explicitly

1. The timestamp that means nothing. updated_at set by an ORM onUpdate hook, so it records when a row was written rather than when the value was true. Every freshness computation downstream is then measuring your own write traffic.
2. Clock skew as data corruption. A supplier's as_of five minutes in the future makes age negative, which makes everything look eternally fresh. Clamp, count the clamps, and alarm if the count grows.
3. Retry storms disguised as demand. A supplier slows down, your retries multiply, their rate limiter trips, everything becomes retries. Full jitter and a shared bucket, or this happens on your worst day.
4. The delta stream that silently stopped. No errors, no alerts, just a cursor that stopped advancing two days ago. Only a silence alarm catches this.
5. Negative caching a timeout. "I could not ask" gets stored as "no," and a brief outage turns into an hour of phantom unavailability.
6. The unconditional upsert. A replayed message reinstates old state. Conditional writes on a monotonic version, always.
7. Uniform reconciliation sampling. Drift measured almost entirely on parts nobody reads, producing a reassuring aggregate that says nothing about the head of the distribution.
8. Freshness in a header. Stripped by a proxy, dropped by a client wrapper, and now every consumer treats absence as fresh.
9. A composite that averages. A quote assembled from four supplier facts is exactly as fresh as its oldest input, and reporting the mean makes stale data look acceptable.
10. Descriptive fields on the hot path. Re-polling immutable specifications every fifteen minutes, then wondering where the rate limit went.

A compressed incident, and what it teaches

Sketch of the shape these things take, offered as an illustrative scenario rather than a report of a specific event.

Mid-October, a chinook collapses overnight and the forecast flips. Search volume for winter sizes triples inside twelve hours, which is the demand-side load spike anyone doing seasonal changeover work plans for. The hot tier of the polling scheduler swells accordingly, the shared token bucket starts throttling, and warm-tier items begin aging past budget.

Meanwhile — and independently — one distributor's delta stream stalls at 04:00. The nightly snapshot had run at 02:00, so every record looks recent enough. Availability continues to be served from a snapshot that is now nine hours old on the most volatile field in the system, during the busiest demand hour of the season.

What surfaces the problem is not the polling metrics, which look healthy. It is two things. The silence alarm fires because a stream that normally emits thousands of changes emitted forty. And the reconciliation loop's drift rate on hot-tier availability jumps from a fraction of a percent to double digits — records wrong while inside budget, which is precisely the signal that says "your inference is broken, stop trusting it."

The lesson is not "add more polling." It is that the two mechanisms that caught it were both mechanisms that check the system against reality instead of against itself. Poller dashboards measure whether your code ran. Drift and silence measure whether your code was right. Only one of those pairs is worth waking up for.

What to build first

If you are retrofitting this into an existing integration, the order matters, because each step makes the next one measurable.

Start with the three timestamps and a read log. You cannot argue about budgets without knowing your current age distribution, and most teams are startled by their own p99. Add the change log next; two weeks of it gives you real rates. Then write budgets down — badly at first, refined by the drift signal later — and put them behind a policy function rather than scattering constants.

After that, the commitment path: make binding decisions revalidate synchronously, and make the type system enforce it. That single change removes the most expensive category of error before you have optimised anything. Conditional requests come next, because they make everything else affordable. Reconciliation and the silence alarm after that. Adaptive scheduling last — it is the most fun to build and the least valuable until the measurement layer exists to tune it.

Underneath all of it is one commitment worth restating plainly. You are never going to know the true state of somebody else's warehouse. You can know how old your information is, how fast that kind of information decays, and what you are willing to risk at each decision point. A system that knows those three things and says so out loud is more trustworthy than one that hides an unknown behind a confident boolean — whether the consumer is a rendering engine, a fleet planner, or a person at a counter asking whether their existing set can be repaired or whether new rubber needs to be sourced and balanced this week.

Freshness is not a feature you add. It is a number you owe your users, and the only way to owe it honestly is to write it down first.