In the first half of this I took apart what a server sees before it reads a single HTTP header: the cipher list, the extension set, the protocol your client asks for, the HTTP/2 settings underneath. My Python script announced itself in four separate ways and then claimed in a header to be Chrome, which is roughly as convincing as a forged passport with the photo still in the plastic.
This half is the experiment. Same 500 domains as before, same three servers in New York, Frankfurt and Singapore, three clients that differ only in how they introduce themselves.
The first is honest Python, sending Python-urllib/3.12 as its user agent and making no attempt to be anything else. The second is the client from the previous study, Python's urllib wearing a current Chrome user agent, which is what most scripts in the world look like. The third sends Chrome's actual TLS handshake, the same fifteen cipher suites in the same order, the same extensions, HTTP/2 with Chrome's settings and pseudo-header ordering, along with the same user agent string.
Everything else is identical. Same machines, same moment, same URLs, one request each, a small pause between them so as not to be rude.
The numbers
Counting only requests that received an HTTP response at all, and pooling the three regions:
| client | refusal rate | refusals |
|---|---|---|
| honest Python | 35.1% | 405 of 1,155 |
| Chrome user agent, Python handshake | 26.9% | 312 of 1,159 |
| Chrome handshake and user agent | 20.2% | 235 of 1,161 |
Changing a single header string removed about a fifth of the refusals. Changing the handshake underneath it removed a quarter of what was left. Between the two, four in ten of the doors that had been closed opened.
The pattern holds in each region independently, which matters because it means I am not looking at one unlucky address range:
| region | honest | UA only | full fingerprint |
|---|---|---|---|
| New York | 33.0% | 25.3% | 18.2% |
| Frankfurt | 34.4% | 25.4% | 17.7% |
| Singapore | 37.8% | 30.1% | 24.7% |
Which sites changed their mind
Eighteen domains refused the user-agent-only client in all three regions and served the one with Chrome's handshake in all three regions. Not sometimes, not in one city. Every time.
facebook.com whatsapp.com whatsapp.net wa.me
fb.com fbcdn.net live.com wordpress.com
wp.com cisco.com comcast.net webex.com
w3.org forms.gle pexels.com mediafire.com
chaturbate.com stripchat.com
Exactly one domain went the other way and refused the better-disguised client, which I assume is noise rather than a site that specifically dislikes Chrome.
I want to sit on w3.org for a moment. The World Wide Web Consortium publishes the specifications for HTTP and TLS. Its web server declined to serve a program that implemented those specifications correctly using the Python standard library, and served the same program once it had copied Chrome's handshake byte for byte. Nothing about that is malicious or even unusual, it is a bot rule somewhere in front of the site doing what bot rules do, but it is a good illustration of how the defaults have settled.
What did not change
A fifth of the sites refused the fully disguised client anyway, and that number deserves as much attention as the part that moved.
The experiment before that established why: fourteen sites served my home broadband while refusing every datacentre I tried, and the pattern of who got through varied with which address range I happened to be handed rather than with any national boundary. Fingerprints are one filter. Address reputation is another, sitting behind it, and no amount of handshake mimicry gets you past a list your IP is already on.
Singapore is the clearest illustration. With the full Chrome fingerprint it still had a 24.7% refusal rate against 17.7% in Frankfurt, using identical software against identical domains at the same time. The only difference is the addresses those droplets were allocated from.
The uncomfortable part
I should be straight about what I have actually demonstrated, because there is a reading of this that makes me uneasy.
I have not found a clever trick. Everything above uses a library that exists for exactly this purpose, takes one argument, and needed no understanding of TLS to operate. The measurement is honest, the client sent one request per domain and identified itself as Chrome because that is what the experiment required, and I would rather publish the number than pretend the technique is secret. It is documented, packaged and a single pip install away.
What the number shows is that a large amount of the web's defensive tooling is currently sorting on a signal that is cheap to forge. Anyone scraping at volume already knows this and has known for years. The people it actually stops are the ones who were never the problem: a feed reader, a link preview service, a university crawler, an accessibility tool, someone's hobby script. Those get the 403. The commercial operation with a budget copies a browser fingerprint and does not.
That is the part I would want a site owner to take from this rather than the technique. If your bot rules are tuned to TLS fingerprints and user agents, the population you are excluding is not the population you think.
What I would do next
The obvious follow-up is to measure the other direction. Every number here is about how servers treat clients, and the honest limitation is that my Chrome impersonation is very good but not perfect. Its cipher list matched real Chromium exactly, its HTTP/2 fingerprint matched exactly, and the extension count was off by one, which is a difference a determined detector could use.
So the version of this study I have not run is the one where the client really is a browser, driven from all three regions, on the same domain list at the same hour. That would separate "refuses non-browsers" from "refuses this very good imitation of a browser", and the gap between those two numbers is the actual size of the fingerprinting industry's advantage.
The whole thing cost about four cents in droplets. Three of the cheapest machines DigitalOcean rents, created through the API, alive for roughly forty minutes each, destroyed afterwards. The reason I keep noting the cost is that this kind of measurement reads as though it needs infrastructure, and it does not. It needs three small machines in different places and the patience to run the same list three times.
