"The site is behind Cloudflare, can you still scrape it?" is the most common question Apex Automation Team gets in feasibility tests. Usually yes — but the answer depends on how the site is configured and how respectfully the scraper behaves.
How bot protection decides
Modern protection scores each request on TLS and HTTP fingerprints, browser signals (JavaScript execution, fonts, WebGL), IP reputation and behaviour over time (request rate, navigation patterns). Plain HTTP libraries with default headers fail the first check; a real browser at human speed from a clean residential IP usually passes.
What works
- Real browsers: Playwright/Puppeteer with a consistent, realistic fingerprint.
- Clean IPs: residential or ISP proxies with low concurrency per IP.
- Human pacing: randomised delays, scroll and navigation patterns instead of hammering endpoints.
- Session reuse: pass the challenge once, then reuse cookies for the session's lifetime.
- Hidden APIs: once a session is established, the site's own JSON endpoints are often reachable with the session cookies — far more efficient than rendering pages.
What does not work (or should not be done)
Datacenter IPs at high concurrency, captcha-farm abuse, and anything that violates the site's terms or bypasses authentication. If a site clearly prohibits automated access, we tell the client and look for an official API, a data partner or a different source.
Our approach
Every project starts with a free feasibility test: we measure block rates at safe concurrency and report the realistic throughput and cost. That number goes into the fixed quote. See custom web scraping and API extraction. Not legal advice — laws and terms vary.
Quick checklist
- Check for an official API or data partner first
- Test block rates at low concurrency before scaling
- Use real browsers and residential IPs for protected pages
- Reuse sessions and look for hidden JSON endpoints
- Stop and reassess if the site's terms prohibit automated access
Frequently asked
Do you solve captchas?
We design scrapers so captchas are rarely triggered — human pacing, clean IPs, session reuse. We do not bypass authentication or abuse captcha farms.
What throughput is realistic on a protected site?
Typically hundreds to a few thousand pages per hour at safe concurrency; the feasibility test gives an exact number for your target.