A scraper that only produces a CSV is half a product. Clients want rows to appear where the team already works. Apex Automation Team designs the delivery pipeline as carefully as the scraper itself. The pattern we use most:
1. Normalise at the source
The scraper emits clean, typed records with a stable unique key (seller ID, URL hash, SKU). Everything downstream relies on that key for deduplication.
2. Choose the transport
- Google Sheets: simplest; append-only sheets with a "last updated" column. Apps Script handles formatting and alerts.
- Webhooks to Zapier/Make: one event per record; Zapier/Make fan out to CRM, Slack, email tools.
- Direct HubSpot/ClickUp API: for volume and reliability, the scraper upserts contacts/companies/tasks itself, with property mapping and rate-limit handling.
- Database/S3: for analytics teams; Parquet/CSV drops with a manifest.
3. Dedupe and update, do not duplicate
Upserts keyed on the unique ID; changes recorded as updates rather than new rows. For outreach tools (Smartlead, Prospeo syncs) suppression lists are checked before a contact is added.
4. Retries, logs, alerts
Every delivery is retried with backoff; failures land in a dead-letter sheet and trigger a Slack/email alert with the record and the error. A daily summary shows counts by status.
5. Approval gates
For anything that emails or spends money, a human approval step (a checkbox column, a ClickUp status) sits between the pipeline and the action.
This is the same architecture behind our multi-platform business automation projects (ClickUp · Apps Script · Zapier · Smartlead · HubSpot). See workflow automation or tell us where the data should land.
Quick checklist
- Give every record a stable unique key
- Upsert, never blindly append
- Add retries, a dead-letter sheet and alerts
- Put an approval gate before emails or spending
- Log counts per run for a daily summary
Frequently asked
Zapier or Make?
Both work; Make is cheaper at volume and better for branching, Zapier has broader app coverage. For high volume we call HubSpot/ClickUp APIs directly.
Can you connect to a tool you have not worked with?
Almost always — if it has an API or webhooks, we integrate it.