projectdiscovery/katana

Katana is a fast web crawler and spidering tool built for security researchers and automation pipelines. A web crawler is a program that automatically visits web pages and follows links to map out a website, much like how search engines index the internet. Katana goes further than basic crawlers by also handling JavaScript-heavy websites, which many older tools miss because they only look at the initial HTML without running the scripts that build the actual content. With nearly 17,000 stars on GitHub, it is a popular tool in the security and bug-bounty community. It offers two modes: standard mode for quick crawling of regular websites, and headless mode that launches a real browser under the hood to handle sites that rely heavily on JavaScript. You can control how deep it crawls, set time limits, filter which parts of a site to include or exclude, and have it automatically fill out forms to discover more of a site. Output can go to the terminal, a file, or JSON format for easy piping into other tools. It also supports proxies, custom HTTP headers, and resuming interrupted scans. Installation is via the Go package manager or Docker. It is primarily a command-line tool aimed at developers and security professionals who need to systematically map out a website or web application.