What I learned separating daily collection from weekly interpretation in a cron pipeline

The question came from a reasonable place: if market-listening sweeps only run once a week, are meaningful day-over-day changes getting missed? I spent a few hours measuring before answering, and the measurement led to an architectural split I hadn't planned on.

The answer: collect daily, interpret weekly. Not because weekly is the right cadence for collecting, but because the instrument I'm querying doesn't have daily resolution.

The instrument's resolution problem

YouTube search ranking sounds like a high-resolution signal. A video that gained 10,000 views overnight should move in the results the next day. In practice, the ranking surface is more like a weather map: high-resolution in principle, but what you see at any moment is partly the weather and partly display non-determinism.

I measured this by fetching the same five search queries, then fetching them again roughly 60 seconds later, and comparing the two result lists using Jaccard similarity. A Jaccard of 1.0 means the lists are identical; 0.0 means no shared entries.

The same-minute refetch returned Jaccard similarity between 0.43 and 0.88. A genuine two-day difference — comparing two days' fetches of the same query — returned 0.03 to 0.58.

The same-minute measurement noise (0.43–0.88) substantially overlaps with the actual two-day signal range (0.03–0.58). When the noise and signal ranges aren't separated, you can't reliably attribute a change to "something changed in the real world" versus "the ranking shuffler surfaced different items." View counts have the same problem: across two consecutive daily fetches, 59% of view counts were byte-identical. The changing ones had mostly crossed a display-rounding threshold ("1.2K" vs "1,247"), not a real change in views.

Conclusion: this instrument has weekly resolution for persistent trends, not daily resolution. Running the sweep daily doesn't change that.

Why daily interpretation means publishing noise as strategy

Four downstream systems read the shared market-trends.md file: the YouTube Short generation routine, the Bluesky post queue, the article routine, and the market monitoring sweep. I described how that file works as a shared context hub in detail last week. The Bluesky routine reads its FLAT/CHANGED columns directly into subject selection.

When market-trends.md was updated daily, single-fetch noise propagated immediately into Bluesky subject selection. A video that appeared in the top-15 because the shuffler surfaced it would become a "trending subject" for the next day's posts. Those posts would go out on a subject with no genuine search demand — just a one-day shuffler artifact.

The failure was invisible. The posts looked normal, the subjects had the right category, nothing in the logs flagged a problem. The degradation was slow drift toward subjects with no measured demand. This is the category of failure that's hardest to catch: the pipeline runs, the outputs look plausible, and the error accumulates silently. Recognizing the shape of that failure is what the pipeline health monitor pattern is designed to surface — but only if you've defined what "healthy output" means. I hadn't.

Three design rules in the new collection layer

The fix was a dedicated daily collection layer with no interpretation step. scripts/market-listening/collect.mjs runs at 17:00 UTC every day and writes to data/market-listening/YYYY-MM-DD.json. Three rules in the code are treated as non-optional.

Rule 1: Repeat-and-vote. Each query is fetched three times. A video only makes the output if it appears in at least two of the three fetches. The constants are REPS = 3 and MIN_APPEARANCES = 2.

// Repeat-and-vote parameters. 3 reps / 2 votes is the cheapest setting that
// removes single-fetch flukes: at Jaccard ~0.6 a genuine top result appears in
// all 3 reps, a shuffler artifact typically in 1.
const REPS = 3;
const MIN_APPEARANCES = 2;

At a Jaccard of ~0.6, a video that genuinely ranks in the top 15 appears in all three fetches. A shuffler artifact that randomly surfaced once might appear in one fetch, rarely in two, almost never in all three. In live dry-runs before launch, this rule dropped between 16% and 36% of unique videos per query. Over the first full collection run on 2026-08-12, 86 of 287 unique candidate videos were filtered as noise. That's more than a quarter of the raw output — gone before it reached the interpretation layer.

Rule 2: Never swallow a failure. The previous trends-fetch workflow used a fetch helper that caught HTTP 403 errors and returned an empty array. Two Reddit sources went dead and nobody noticed for 71 days because the collection file kept being written and kept looking plausible. Empty arrays aren't obviously wrong; a file with fewer signals just looks like a quiet week.

In the new script, every HTTP failure is written to an errors[] array in the output JSON and the matching entry in sources_ok is flipped to false. The committed file itself is the alert:

{
  "sources_ok": { "youtube": true, "reddit": false, "autocomplete": true },
  "errors": [
    { "source": "reddit/game-recs", "detail": "HTTP 403", "at": "2026-08-12T17:14:22Z" }
  ]
}

A consumer reading the file sees immediately that reddit-sourced signals are absent. The same principle drives the GitHub Issues monitor — make failures visible in the committed artifact, not just in runner logs that nobody reads daily.

Rule 3: Parsed JSON only, never raw HTML. The initial collection prototype wrote full YouTube search HTML to disk for later parsing. A single full sweep was 15MB. Annualised, that's ~5.5GB of git history for one workflow. The parsed-rows-only format runs ~98KB per day — about 150× smaller. The lessons from managing GitHub Actions artifact storage apply here: storage costs are real, but reviewability is the bigger reason. A 98KB JSON diff is readable when something goes wrong; a 15MB HTML diff is not.

The interpretation layer stays weekly

The collection script has one explicit comment that appears twice:

// This script never interprets, it only measures.

The weekly interpretation sweep reads multiple consecutive days of collection output and looks for two types of signal above the noise floor:

signals.persistent_new: a video that appeared in the top-15 for three consecutive daily collection runs. Persistence across three independent days of REPS=3 collection means this video appeared in 2-of-3 fetches for three days straight — 6 of 9 possible observation slots. That's a much stronger signal than any single-day reading, and it has a measured false-positive rate (the 86/287 noise drop gives an upper bound).
signals.autocomplete_shift: YouTube's autocomplete suggestions change slowly — on a days-to-weeks timescale rather than minute-to-minute. Day-over-day autocomplete changes are above the noise floor where ranking isn't.

The interpretation sweep still uses an LLM — the same session-coordination pattern from this write-up — but only once a week, and only when the persistent signals give it something real to reason about. A daily LLM interpretation pass over single-fetch noise would produce outputs that look thoughtful and describe the shuffler.

The timing was also designed around conflict avoidance. The auto-tuner lands at 20:42–20:47 UTC in practice. Collection at 17:00 UTC leaves 3.5 hours of clearance. The cron concurrency discipline for this monorepo uses workflow-level groups, so the time separation matters.

What downstream consumers gained

Before the split, the market-trends.md file's updated: date meant "when someone last ran a sweep" — which conflated collection age with interpretation age. A consumer had no reliable way to distinguish stale interpretation from no data.

After the split, three distinct states are visible:

State	Signal	Consumer action
`sources_ok: all true` + recent daily commit	Collection healthy	Use data normally
`sources_ok: youtube=false`	Source broken	Use remaining signals, flag gap
`updated:` older than 14 days	Interpretation stale	Fail-closed to defaults

The freshness gate described in the shared context file write-up now has sharper semantics because collection freshness (daily commit history) is separate from interpretation freshness (updated: date). The pipeline health monitor checks sources_ok flags on every run and would open a GitHub Issue if both YouTube sources failed simultaneously — impossible to detect before the split.

What I'd do differently

Measure the instrument's day-over-day resolution before designing the collection frequency. The Jaccard measurement took about two hours. I spent a month assuming daily sweeps produced daily-resolution signals because the sweep ran daily. Those are not the same thing.

The middle ground I considered — keeping daily interpretation but applying a 7-day rolling average to raw scores before writing the file — wouldn't have worked. Averaging noise doesn't produce signal; it produces a smoother representation of noise. The Jaccard non-determinism problem is that individual observations aren't reliably measuring the underlying ranking. Averaging them gives you a smoother non-signal, not a real one.

The correct insight is that you need consensus across independent samples taken at the same moment (the REPS=3 vote), not smoothing across repeated moments. Voting filters out non-signal at observation time; no amount of post-hoc averaging replaces that.

FAQ

Why 3 fetches specifically?

At Jaccard ~0.6, a genuine top-15 result appears in all three fetches. A shuffler artifact typically appears in one, rarely two, almost never all three. The 2-of-3 threshold is the cheapest configuration that removes single-fetch flukes while keeping collection under 3 minutes (5 queries × 3 reps with 5–15s jitter between hits). REPS=5 would have pushed runtime past 5 minutes and tripled the request count to YouTube's servers.

What's autocomplete_shift and why is it above the noise floor?

YouTube autocomplete returns suggested completions for seed prefixes ("stardew valley vs...", "best indie game..."). These suggestions update on a days-to-weeks timescale rather than minute-to-minute. When the autocomplete response for a seed changes between two consecutive daily fetches, that indicates genuine search-demand movement — not ranking non-determinism.

Doesn't the 17:00 UTC cron conflict with other writes?

The auto-tuner lands at 20:42–20:47 UTC in practice, giving 3.5 hours of clearance. Earlier attempts at 20:00 UTC caused occasional push conflicts. The collection script's push step includes a 3-attempt retry with git pull --rebase on conflict — the same pattern used by other write workflows in this monorepo — but time-based separation is simpler than relying on retry logic.

Could you just interpret daily with a longer rolling window?

The problem isn't window length; it's that each individual observation is unreliable at daily resolution. Averaging seven noisy daily fetches gives a smoother curve but each data point still carries Jaccard noise of 0.43–0.88. The vote removes non-signal at observation time. No amount of post-hoc averaging replaces that.