I scrape data for a living, and most of that job is the same fight over and over: a site has data I need, and it really doesn't want me to have it. This is the story of one wall that beat every trick in the usual playbook — and what finally worked.
The target
I needed a set of listings from a large classifieds site (mobile.de — car listings). Nothing exotic: title, price, a few fields per listing. The kind of thing that should take twenty minutes.
It took a lot more than twenty minutes.
Attempt 1 — plain requests
I started where everyone starts: a plain HTTP request — just requests.get with a normal User-Agent header.
import requests
r = requests.get(url, headers={"User-Agent": "Mozilla/5.0 ..."})
Blocked. Not a 404, not a timeout — a challenge page. And here's the first trap that eats a lot of beginners: the response came back 200 OK. Status says success, but the body is a bot wall, not the data. If your scraper only checks the status code, it thinks it won and happily saves a page of garbage.
Lesson 1: a 200 is not a win. Read the body. A block page returned as 200 OK is a failure, not data.
This wasn't a missing header or a bad IP. The site was running Akamai Bot Manager, and it had already decided I wasn't a browser.
Attempt 2 — look like a real browser (TLS spoofing)
If the problem is "you don't look like a browser," the next move is to look more like one. Not just the User-Agent string — anyone can fake that — but the TLS fingerprint: the exact way a real Chrome negotiates the HTTPS connection (cipher order, extensions, the JA3/JA4 signature). Tools exist to impersonate that.
So I made my requests carry a real Chrome TLS signature.
Still blocked.
That surprised me the first time, and it's the part most guides skip. Matching the TLS handshake gets you past the front door, but Akamai doesn't stop at the handshake. It runs JavaScript in the page that measures things a bare HTTP client simply doesn't have: a real rendering engine, timing, browser internals. No amount of header or TLS spoofing produces those, because there's no browser behind the request.
Lesson 2: TLS fingerprinting is necessary but not sufficient. Modern bot managers check what happens after the connection, inside a real page.
Attempt 3 — headless browser
Fine — if it wants a browser, I'll give it a browser. Headless Chrome via automation (Playwright/Puppeteer).
Closer. But still flagged.
Headless browsers leak. They announce themselves in dozens of small ways — automation flags, missing or inconsistent browser internals, subtle differences a stock headless build carries out of the box. Akamai's script is built precisely to notice those. You can spend days patching each tell one by one, and the wall moves the following week.
Lesson 3: a headless browser is still not a normal browser. The gap is exactly what these systems are trained to detect.
What actually worked — a real browser, driven remotely
The breakthrough was to stop trying to simulate a legitimate browser and just use one.
I launched a real, ordinary Chrome — the same binary a human uses, with a real user profile — and drove it programmatically over the Chrome DevTools Protocol (CDP) instead of launching a fresh automated instance.
In practice: start Chrome with remote debugging enabled, then connect to it with Playwright's connect_over_cdp on the local debugging port.
From Akamai's point of view, there was nothing to catch: it was a real browser, with a real engine, a real profile, real everything — because it literally was. The listings loaded. I extracted 64 complete listings, cleanly.
The difference between attempt 3 and this one is subtle but it's the whole game: a headless/automated browser pretends to be legitimate and leaks. A real browser you happen to be driving is legitimate. There's nothing left to fake.
The lesson under the lesson
Here's the part that matters more than any single technique:
Getting past the wall once is an afternoon. Keeping it working for six months is a different job.
The wall changes. IPs get flagged. What passes today gets caught next month. The real work isn't the clever trick — it's the diagnosis (which wall am I even facing?), the maintenance, and knowing when to stop hammering and change approach entirely. Anyone can copy a snippet. Reading the wall in front of you is the actual skill.
If you're stuck on a wall
If you've got a site that keeps blocking you and you can't tell why — that's the interesting part, and I'm happy to look. Tell me the symptom (403? a 200 that's secretly a challenge? works in the browser but not in code?) and I'll tell you what you're most likely up against.
That diagnosis is free. What you do with it is up to you.
