Quick answer
We pinned our Bayt.com scraper's proxy to the country it scrapes, and that pin was the block. On Apify residential exits in the UAE, cloud QA landed 0 rows in 15 attempts. We changed one field to a UK residential exit and the same build returned 20 rows, then a full 60-row run with no blocked pages. The data stayed genuinely Emirati because Bayt decides the country from the URL path (/en/uae/jobs/...), not from where your IP sits. The Bayt Jobs Scraper now defaults to RESIDENTIAL / GB, and you pick the market with the country input.
Shouldn't a proxy match the country you're scraping?
Usually, yes. A geo-random exit can quietly give you plausible but wrong data: prices in another currency, or another region's catalogue. Pinning the exit is normally the safe default, and it's our fleet rule for exactly that reason. The rule assumes the site uses your IP to pick the country. Bayt doesn't. It scopes every search by a path segment, so /en/uae/jobs/accountant-jobs/ returns UAE jobs whatever exit the request comes from. That leaves the pin with a cost and no benefit. The UAE residential pool was the one Bayt's Cloudflare layer challenged hardest, so pinning to it made us look more suspicious.
How do you tell "the pin is the block" apart from "the site blocks us"?
Change one variable. Same build, same input, same residential tier, and only the exit country different: AE went 0 for 15, GB cleared on the first attempt and kept clearing over three spaced runs. Swapping tiers or TLS profiles at the same time would have hidden which change did the work. Before you shelve a target as unreachable, try a neutral exit.
If the site picks its country from the URL, the exit country is just one more fingerprint, and the local one can be the most-watched.
What the Actor gives you
One row per job: job_id, title, company, company_url, location, career_level, salary (only on postings that publish it), a remote flag, summary, posted_at_raw (e.g. "3 days ago"), the absolute detail_url, and the keyword that found it. Pass several keywords in one run, and set country to uae, saudi-arabia, egypt or any other Bayt market path.
Honest limitations 🚧
List-page fields only, with no follow-through to detail pages. The posted date is Bayt's own relative text, kept verbatim rather than guessed into a timestamp. Salary appears on a minority of postings, and we return null rather than invent it. A search with no matches is a successful run.
FAQ
Do I need a Bayt account?
No. It reads Bayt's public search result pages.
Why a UK proxy for UAE jobs?
Because Bayt picks the market from the URL, and UK residential exits cleared where UAE ones were challenged. The rows are still the UAE (or whichever country you set).
What about blocks and captchas?
That's our job. We rotate browser fingerprints, retry with backoff on rate limits and challenges, and rotate residential sessions on every block.
$0.20 per run plus $0.002 per job row: $2.20 per 1,000 results. A zero-match search succeeds and costs only the start fee.
Built by Devil Scrapes. We handle the fingerprints, the proxies and the retries, and we test the proxy default we ship against the target before a customer ever runs it.
