Smashing a hidden I/O bug: 100 disk reads per run for data that never changed

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

awesome-tech-newsletter is an open-source project by Infrasity Labs that auto-discovers tech newsletters across the web. It runs a fleet of Python fetchers — for Substack, Hacker News, Medium, Product Hunt, Beehiiv, Hashnode, and more — that crawl each platform, classify what they find against a shared keyword config, and aggregate everything into a curated, categorized directory in the README.

Bug Fix or Performance Improvement

I picked this off the project's open issue tracker: issue #41, a performance bug in the Product Hunt fetcher.

The fetcher pulls 100 recent posts from the Product Hunt GraphQL API, then loops over them to classify each post against a list of keyword queries. The problem was this line, sitting inside the per-post loop:

for post_edge in posts:
    # ...build text_corpus...
    queries = get_search_queries(append_newsletter=False)  # <-- every iteration
    for query, cat in queries:
        ...

That looks like a harmless list rebuild — until you read what get_search_queries() actually does:

def get_search_queries(append_newsletter=True):
    config_path = os.path.join(..., "config.json")
    with open(config_path, "r", encoding="utf-8") as f:   # disk read
        categories = json.load(f)                          # JSON parse
    # ...build query list...

It opens and JSON-parses config.json from disk on every single call. So every fetch run was doing 100 file opens and 100 JSON parses for data that cannot change mid-loop.

The smoking gun that this was a bug and not a design choice: every other fetcher in the repo (hackernews.py, substack.py, medium.py, and six more) calls get_search_queries() exactly once, outside its loop. Only producthunt.py did it per-post.

Measured impact (benchmarked against the repo's real config.json — 9 categories, 86 generated queries — with 100 simulated posts):

Honest framing: this is a scheduled crawler, so no user was staring at a spinner. But the cost scales with post count × config size, and redundant loop-invariant I/O is exactly the kind of silent tax that compounds as a project grows.

Code

PR: https://github.com/Infrasity-Labs/awesome-tech-newsletter/pull/43

The full change:

             posts = data.get('data', {}).get('posts', {}).get('edges', [])

+            # Load search queries once: get_search_queries() reads and parses
+            # config.json from disk on every call, and its result never changes
+            # within a fetch run, so calling it per-post did 100 redundant
+            # file reads + JSON parses.
+            queries = get_search_queries(append_newsletter=False)
+            seen_urls = set()
+
             for post_edge in posts:
                 node = post_edge.get('node', {})
@@
-                queries = get_search_queries(append_newsletter=False)
-                for query, cat in queries:
-                    if query in text_corpus:
+                for keyword, cat in queries:
+                    if keyword in text_corpus:
                         is_tech = True
                         category = cat
                         break
@@
-                    if not any(d['url'] == target_url for d in discovered):
+                    if target_url not in seen_urls:
+                        seen_urls.add(target_url)
                         logger.info("Discovered Product Hunt: %s", target_url)

My Improvements

Three changes, in decreasing order of importance:

1. Hoisted the loop-invariant call. get_search_queries() now runs once per fetch, right after the API response is parsed. This turns 100 disk reads + JSON parses into 1. I deliberately did not add caching inside get_search_queries() itself — that would change behavior for every fetcher and risk serving stale config in long-lived processes. Fixing it at the single bad call site keeps the change surgical and matches the pattern the other nine fetchers already use.

2. Fixed a variable-shadowing landmine. The inner loop used for query, cat in queries: — but query was already the name of the GraphQL query string defined earlier in the same function. It happened to be harmless today because the GraphQL string isn't reused after the request, but it's the kind of shadowing that turns a future "add retry logic" PR into a mystery bug. Renamed the loop variable to keyword.

3. Replaced an O(n²) dedup with a set. The duplicate-URL check was any(d['url'] == target_url for d in discovered) — a full list scan for every discovered post. A seen_urls set makes it O(1) per lookup. Minor at 100 posts, but free to fix while I was in the function.

Verification: the module compiles clean (python3 -m py_compile), imports, and runs correctly through its no-token code path. The benchmark hooked builtins.open to count file accesses, confirming the 100 → 1 drop, and used timeit over 20 repeats for the timing numbers.

Transparency note: I found, benchmarked, and fixed this bug working alongside Claude (Anthropic's AI assistant) — it ran the code analysis and benchmarking in a sandboxed environment while I directed the hunt, reviewed the changes, and submitted the PR.