Scrapy + BigQuery Data Pipeline

The problem

The client needed structured data on damaged cars from a listings website that offered no proper API. The list of pages to scrape already lived in BigQuery, and the extracted records had to land back in BigQuery, so a one-off scrape or a CSV export was not an option. The scraper had to be a service that runs continuously and keeps the tables fresh.

What I built

• A Scrapy spider in Python that crawls the damaged-car listing pages and extracts the fields the client needed.
• A BigQuery input stage: the spider reads its target URLs from a BigQuery table instead of a hard-coded list.
• Scrapy item pipelines that take the scraped items and push them into several BigQuery output tables.
• A deployment of the Scrapy bot to the cloud so it runs 24/7 rather than from a laptop.

Data and infrastructure

BigQuery was both the source and the sink. Keeping the URL queue in a table meant the list of targets could change without touching the code, and writing the results straight into BigQuery meant the data was queryable the moment the spider finished a page. Running the bot in the cloud removed the need for anyone to start it by hand.

Outcome

A BigQuery-in, BigQuery-out scraping pipeline that runs around the clock in the cloud and extracts damaged-car listings from a site with no usable API. The client gets fresh, queryable data in the tables they already use, with no manual steps.