Large-Scale Web Scraping on AWS (Scrapy, EC2, S3)

The problem

The client needed data from millions of indeed.com pages, not a one-off sample, and needed it landing in an AWS S3 bucket under a specific folder structure and naming convention so other tooling could pick it up. At that volume two things break: blocking, because a single IP hitting a site millions of times gets cut off, and operations, because a crawl that only runs when someone remembers to start it is not a data pipeline.

What I built

• A crawler on Python's Scrapy framework, written for the structure of Indeed's listing pages.
• Crawlera as the proxy rotator, so requests come from a changing pool of addresses and the bot does not get blocked.
• A custom Scrapy middleware that pushes each scraped item into S3 as it is produced, writing to the folder and file-naming scheme the client specified.
• Deployment onto an AWS EC2 Ubuntu instance, with the spider running on a specified schedule rather than by hand.

Data and infrastructure

EC2 provides the always-on machine, the schedule starts each run, Crawlera handles the network side, and S3 holds the output under the agreed structure. Because the folder and naming convention were agreed up front, the data was ready for downstream use directly from the bucket.

Outcome

The pipeline extracted millions of data points from Indeed and stored them in S3 under the agreed structure, running on schedule from EC2 without being blocked. It is a small system, but it is the shape most production scrapers end up with: a framework crawler, a proxy layer, a storage middleware and a scheduler on a cloud host.