Free proxy lists look generous. Thousands of addresses, updated every few minutes. The first time I tried them for a scraping job, almost nothing connected. The second surprise came later: some of the ones that did connect were not doing what I thought.
This is what I found while building proxy-scraper, an open-source tool that collects public proxies and checks them properly.
Dead is the easy part
Every run starts with about a million candidates from 700+ public lists. Most of them never finish a TCP handshake. That's expected and cheap to detect.
The interesting failures come after that.
Honeypots answer the question you asked
The usual check is: connect through the proxy, ask a "what is my IP" service, see a foreign IP, done. Some proxies answer exactly that request with a plausible "200 + IP" and fail everything else. In some runs they were five out of six "hits".
The fix is a second, independent request. The proxy has to fetch a different site and show the same exit IP there too. Honeypots fall out here, and the same response tells you which headers arrive at the target, which gives you the anonymity level for free.
Some proxies edit what you download
This is the one that surprised me. I added a check that fetches a static HTML page through the proxy and compares its hash with the page fetched directly over verified HTTPS.
Out of 270 proxies that had passed the handshake and the honeypot check, 54 returned a modified page. Almost always an injected <script src="http://…">, sometimes ads. That's 20% of the pool that looked perfectly fine by every usual measure.
If you scrape through free proxies, a fifth of your responses may be rewritten on the way. If you browse through them, you're running someone else's JavaScript.
Where the rest exits from
With the free DB-IP ASN database the tool looks up the provider of every exit IP. In today's list, 52% of working proxies exit from datacenters, which many sites block on sight. And 28% of exit IPs are on the SpamCop blocklist, which means captchas on a lot of sites. Both can be filtered out.
Learning which lists are worth it
A proxy that worked in the last run is far more likely to work now than a random entry. So proxy-scraper checks known-good proxies first, then proxies from sources with a good hit rate, and it drops lists that went stale. Unchanged lists are skipped with ETags: a second run right after the first loads 0 MB instead of about 160 MB.
Try it
pipx install git+https://github.com/maximilianfeix/proxy-scraper.git
proxy-scraper --want 50 --https-only
--serve turns the result into a rotating proxy (HTTP and SOCKS5 on one port, sticky sessions, country per request through the username, optional password, refills itself in the background), there's a Python API, and -o - prints the hits for pipes. If you don't want to run anything, a GitHub Action checks everything every hour and publishes the list: maximilianfeix.github.io/proxy-scraper
The code is MIT licensed. If you find a check I'm missing, issues are very welcome: github.com/maximilianfeix/proxy-scraper
Free proxies are run by strangers. Never send passwords or personal data through them.
