Open SourceScraping14k
Katana
A fast crawler built for security recon and URL discovery.
Katana is a fast, scalable web crawling framework from ProjectDiscovery, written in Go, designed for thorough URL discovery via both standard and headless JavaScript-aware crawling. It supports scope control, form filling, passive sources, and custom field extraction with a focus on automation pipelines. It is notable in the security and recon space for its speed and configurable crawling depth.
Repository
projectdiscovery/katanaLanguage
GoWhat you'd build with it
- Enumerating a site's full URL surface before scraping or analysis
- Crawling JS-rendered apps to discover hidden endpoints and assets
- Feeding discovered URLs into automated data or security pipelines
Tags
crawlingrecongourl-discovery