My stock research pipeline ingests prices and financials for every listed company in Japan, runs event-study backtests, and tracks a live paper portfolio — on delayed free-tier data, a laptop, and an infrastructure bill of exactly zero yen. This is the architecture, and the design decisions that made "free" actually work.
When people hear "financial data pipeline," they picture streaming infrastructure and four-figure data subscriptions. My entire setup is: one official free API, Python, parquet files, and a macOS scheduler. It has processed 276 small-cap stocks and 1,385 earnings events through backtests, and it runs unattended every trading day.
The unlock wasn't a clever tool. It was accepting one constraint and designing around it.
1. Why delayed data + a free tier is actually practical
My data source is J-Quants, the Japan Exchange Group's official API, whose free tier delays data by 12 weeks. That sounds disqualifying — until you separate what a research pipeline actually does into two jobs:
- Research (backtesting, screening, factor analysis): operates on history. A 12-week delay is irrelevant when you're studying six years of earnings events.
-
Execution (what's the price right now?): needs today. But it needs today for only ~20 tickers in my portfolio, not the whole market — and a free quotes library (
yfinance) covers that.
Keeping these two concerns in separate modules means the expensive requirement (fresh data) shrinks to a tiny surface area, and the bulky requirement (deep history, full market) lands on the free tier. Most hobby-quant projects die by paying real-time prices for a research problem.
2. Ingestion: assume the API hates you
The retrieval layer makes three assumptions that turned out to be load-bearing:
-
Responses are paginated — follow the
pagination_keyuntil it's gone. - You will get rate-limited — exponential backoff on HTTP 429.
- Even polite clients should throttle — a minimum 250ms between requests, on top of backoff, because a full-universe pull is thousands of calls and you want to be a good citizen of an API you depend on.
I walked through the client code in the J-Quants guide, so here I'll just note the architectural point: ingestion is its own layer with no business logic in it. It takes an endpoint and parameters, returns rows, and knows nothing about stocks. Every time I've blurred that boundary in past projects, changing a research question forced me to touch (and break) download code.
3. The cache layer: misses are data too
Everything the API returns is written to parquet, keyed by request. The interesting decision — the one that halved my backtest runtime — was what to do when the API returns nothing:
def fetch_cached(key: str, fetch_fn):
path = CACHE_DIR / f"{key}.parquet"
if path.exists():
df = pd.read_parquet(path)
return None if df.empty else df # a cached "nothing" is an answer
df = fetch_fn()
df.to_parquet(path) # persist even when empty
return None if df.empty else df
Financial data is full of legitimate emptiness: delisted tickers, market holidays, companies that didn't exist yet, quarters with no disclosure. My first backtest run spent roughly half its wall-clock time re-asking the API questions whose true answer was "nothing," every single run. Caching the misses — persisting empty frames instead of only successful payloads — cut runtime in half, with zero change to results.
The general lesson: "no data" is a fact about the world, and facts are cacheable. Most caching tutorials only cache hits.
4. Incremental updates — and what tools like Airbyte formalize
With a cold cache, a full-universe pull takes a long evening. With a warm one, the daily update is minutes, because the pipeline only fetches what it doesn't already have: the cache key encodes (endpoint, ticker, period), so yesterday's history is never re-downloaded, and each day appends only the new slice.
That's an incremental sync — hand-rolled. If you've used a data-integration platform like Airbyte, you'll recognize everything above as things it formalizes: sources with pagination and rate-limit handling, destinations, cursor fields ("what did I fetch last?"), incremental vs. full-refresh modes. Building it by hand taught me exactly what those abstractions are for — and where my version is weaker (my "cursor" is a filename convention; state lives in a directory listing; there's no schema migration story). If this pipeline ever outgrows one laptop, the migration path is clear: my ingest layer becomes a declarative source config, parquet becomes DuckDB or Postgres, and the cache logic I hand-wrote disappears into the platform's sync engine. The architecture translates one-to-one — because it's the same problem.
For a personal-scale project, though, the file-based version has a real virtue: you can ls your state. Debugging a sync issue is open cache/8035_daily.parquet, not spelunking through a job orchestrator.
5. Two pitfalls that produce beautiful, wrong results
Both of these produced exciting "findings" that died under inspection:
Stock splits make fundamentals look broken. Splits arrive in the disclosure stream; if you join fundamentals to prices naively, a 1:3 split reads as earnings collapsing by two-thirds — or, joined the other way, as a stock suddenly trading at a third of its "fair value." My event studies had to adjust for split factors before computing any surprise or valuation metric. If your backtest finds that stocks crater on earnings and then recover perfectly, check your split handling before you check your genius.
One-off gains make garbage look cheap. Company-issued forecasts make forward-valuation screens easy — and easily fooled. A company that sold its headquarters building shows a spectacular earnings yield for exactly one year. My screen excludes implausible earnings yields on the theory that if a small-cap looks too cheap, it's usually an asset sale, not a misunderstood business. That single guard removed most of the false positives from the cheapest quintile.
What it all costs
- J-Quants free tier: ¥0
- Quotes for ~20 live tickers: ¥0
- Storage: a few GB of parquet on my laptop: ¥0
- Scheduling: launchd, which macOS already runs: ¥0
- Compute: the laptop I already own
The constraint that made this possible wasn't budget discipline. It was the architectural split in section 1 — once research and execution stopped sharing a data requirement, every remaining piece had a free-tier-shaped hole it fit into.
Whether the strategies this pipeline tests actually make money is a different story — most of mine didn't survive their own backtests — but that's the point of building the pipeline first: cheap, fast falsification beats expensive conviction.
Data is used under J-Quants' personal-use license. Nothing here is investment advice — the pipeline exists precisely because I don't trust my own hunches.
