Scrapy

The battle-tested Python framework for large-scale web crawling.

Scrapy is a fast, mature, open-source framework for crawling websites and extracting structured data at scale, built on an asynchronous Twisted core. It provides spiders, item pipelines, middlewares, autothrottling, and built-in support for selectors, making it the de facto standard for production Python scraping. It is notable for its extensibility and the surrounding ecosystem of plugins and deployment tooling.