Open SourceScraping52k

Crawl4AI

Open-source LLM-friendly web crawler and scraper, built for agents.

Crawl4AI is an open-source Python crawler and scraper built to produce LLM-friendly output, generating clean markdown and structured extractions from web pages. It supports asynchronous crawling, headless-browser rendering, and LLM-based or schema-based extraction strategies. It is notable as a widely starred, free alternative for preparing web data for AI pipelines.

View repo

Repository

unclecode/crawl4ai

Language

Python

What you'd build with it

Crawling pages and converting them to markdown for LLM ingestion
Running async, high-throughput scraping jobs
Extracting structured data via CSS/XPath or LLM strategies for RAG

Repository

Language

What you'd build with it

Tags