Most "impossible to scrape" sites are actually the easiest — once you find the JSON they load themselves. This is the first thing Apex Automation Team does in every feasibility test, and you can do it in five minutes.
Step 1 — Open the Network tab
Press F12 → Network, tick "Preserve log", filter by Fetch/XHR. Reload the page.
Step 2 — Trigger the data
Scroll, click "next page", change a filter. Watch which requests fire. Responses that are JSON with the data you see on screen are your candidates.
Step 3 — Read the request
Check the URL parameters (page, offset, cursor, limit, query), the headers (authorization, x-api-key, csrf tokens, cookies) and the method. Right-click → "Copy as cURL" to reproduce it outside the browser.
Step 4 — Understand pagination and limits
Cursor-based pagination needs the "next" token from each response; offset-based can be parallelised carefully. Note rate limits from response headers or 429 errors.
Step 5 — Tokens and signatures
Some endpoints need a session cookie (log in once, reuse it), a short-lived bearer token (refresh it programmatically) or a request signature computed in JavaScript (reverse-engineer or drive a browser for that one call). This is where API extraction and reverse engineering earns its keep.
Step 6 — Decide
If the endpoint returns clean JSON with reasonable limits: direct extraction, fast and cheap. If everything is signed and rotating: hybrid — a browser establishes the session, the extractor uses it. If there is no endpoint at all: rendered scraping.
Read next: Web scraping vs API extraction. Or send us the URL and Apex Automation Team will do this analysis for free.
Quick checklist
- Filter Network by Fetch/XHR and preserve the log
- Trigger pagination and filters to reveal endpoints
- Copy the request as cURL and replay it
- Note pagination type, limits and required headers
- Check whether tokens expire or are signed
Frequently asked
What if the site uses GraphQL?
GraphQL endpoints are excellent for extraction: one endpoint, structured queries, predictable pagination.
Is using a hidden API allowed?
It depends on the site's terms and the data. We collect only public data and respect rate limits; unsure cases get a frank risk note in the feasibility test.