I built a small Apify actor that watches Certificate Transparency logs for a domain, basically "tell me every SSL cert that's been issued for this domain or its subdomains recently." Useful for catching phishing look-alikes and shadow IT before a customer complaint does. The data source is crt.sh, a free community-run search service over CT log data. Getting the actual query working took ten minutes. Making it reliable took a lot longer, and taught me more about "empty result" ambiguity than I expected from what's basically a single GET request.
First problem: 404 doesn't mean "no results."
crt.sh returns bare HTML error pages, not JSON, when it's under load. I first treated a 404 as "no certificates found," which is a completely reasonable read of a 404. Except I'd query the same domain twice in a row and get a 404 once and a real result the next time. Same query, same data, different outcome, because the 404 wasn't about the data at all, it was crt.sh being overloaded. Same story with 502/503/504. None of these are trustworthy "zero results" signals for this service, so they all need to be retried, not accepted.
Second problem: my retry budget wasn't big enough for a real outage.
I started with 4 attempts and exponential backoff, which felt generous. Then during a bad patch I watched it eat 6 consecutive 502s before a success came through, and the actor's rolling 30-day failure rate had climbed to something like 46%. Four attempts just wasn't enough runway to ride out a real bad stretch on a free community service. Bumped it to 8, and capped the backoff instead of letting it grow unbounded, since a successful response can itself take 10-20 seconds once crt.sh is under load, so the retry loop needs real time budget, not just more attempts fired quickly at each other.
Third problem: a hung connection defeats retries entirely.
Plain fetch() has no timeout. One test run just hung, no 502, no slow success, nothing, indefinitely. If a request never resolves, it never reaches the point where my retry logic would even kick in. Added an AbortController with a per-attempt timeout so a dead connection gets treated as a failure and retried like any other, instead of silently stalling the whole run.
Fourth problem, and the sneaky one: a field just disappeared from the response shape.
The actor filters results by date and sorts newest-first, originally using crt.sh's entry_timestamp field. At some point crt.sh's JSON output for this particular query stopped including that field. No error, no warning, just undefined. Which meant new Date(undefined) >= startDate silently evaluated false for every single entry, so the actor returned zero results, every time, regardless of what data actually existed. The scariest kind of bug for a monitoring tool: it fails by going quiet, not by throwing. Switched to not_before (when the cert becomes valid, which is essentially CT-log time anyway) as the log-time proxy instead, since that field is always present.
None of these were exotic failures. Bare error pages under load, a budget that was too small, a fetch with no timeout, a field that quietly vanished. But stacked together they're the difference between "works when I test it" and "actually trustworthy as a monitoring tool," which matters a lot more once the whole point of the thing is telling you about certificates you didn't expect.
If you want to poke at the actual retry code: github.com/timmKal01/certificate-transparency-monitor. The actor itself is live on Apify if you want to point it at a domain.
