"[Sep 25] Making News Collection Real-Time and Shipping a Standby-GPU Safeguard"

A structural change to news collection solved what a simple time shift couldn't, and a safeguard for running the standby GPU full-time shipped to production today.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

Solving yesterday's leftover problem

Yesterday I shifted the nightly run time to try to get holiday news into the next analysis, but it didn't work — the news-gathering logic had its collection window pinned to the last trading date from the start, so the time shift alone couldn't fix it.

Today I decoupled that collection window from the trading-date cutoff itself. Instead of ending at "the last trading day," the news window now ends at "the run time itself."

I finished the implementation, tests, and an offline dry run. The dry run confirmed the cutoff actually moves along with the run time as intended.

It isn't live yet, though. I'm bundling the deploy with another restart planned later this week, and the first real effect won't be observable for a few more days after that.

I also added a rollback switch — dropping a single marker file reverts to the old behavior instantly if anything goes wrong.

A safeguard for running the standby GPU full-time

Right now one main card handles the core nightly work, and the standby card only gets used for occasional experiments. There's a goal to eventually run the standby card full-time alongside the main one, and yesterday's design review turned up a risky spot.

Part of the existing code handles low-memory situations by broadly clearing related processes, and that pattern could end up killing the standby card's processes while cleaning up after the main job.

Today I rewrote that into a per-card locking structure and shipped it to the resident services. Instead of locking everything as one unit regardless of which card is involved, each card now tracks its own occupancy independently.

The deploy is done, but I still need to watch it hold up under real traffic — the next few days of regularly scheduled jobs running through this new path will be the actual confirmation.

Holding off on the US after-hours snapshot signal

I've been recording a partial overnight snapshot around the time the US market opens, and today I checked whether it was actually worth feeding into the next-day analysis.

The validation came back without a statistically meaningful signal, at either the market level or the per-stock ranking level.

So I'm not injecting it into the analysis pipeline for now — just continuing to record it. I left the door open to re-check once more data accumulates.

Also today

An outside code review went back through recent changes and caught something serious: a safeguard meant to trigger when the standby card misbehaves could, under certain conditions, have also halted the main card's normal work.

Fortunately it was caught before it hit production, and I fixed it right away. The same review turned up a dozen or so lower-priority issues, most of which I addressed too.

I also fixed a noisy-alert issue where a regular timer would fire on market holidays for no reason. I added a holiday check ahead of the relevant jobs, so they now skip quietly on holidays and run normally on trading days.

What's next

Today was about actually finishing the two loose ends from yesterday — the news real-time change and the standby-card safeguard.

Both are freshly deployed, though, so I'll need to watch them over the next week or two to confirm they hold up in production. The US after-hours signal was another case of validating first and deciding to wait.