GPT Crawler

Crawl a site and produce a knowledge file for a custom GPT.

GPT Crawler is a Node.js tool that crawls a website from a starting URL and assembles the extracted content into a single knowledge file ready to power a custom GPT or assistant. It uses Playwright under the hood with configurable selectors, match patterns, and page limits. It is notable for streamlining the path from public documentation to a working RAG-ready knowledge base.