Katana (ProjectDiscovery)

katana -u http://target.com

katana -u http://target.com -d 5

katana -u http://target.com -jc

katana -u http://target.com -kf all

katana -u http://target.com -o urls.txt

Useful Flags

-d     Crawl depth
-jc    Parse JavaScript
-kf    Known file crawling
-o     Output file

Hakrawler

echo https://target.com | hakrawler

cat urls.txt | hakrawler

echo https://target.com | hakrawler -depth 3

Gospider

gospider -s http://target.com

gospider -s http://target.com -d 3

gospider -s http://target.com --js

Feroxbuster (Crawler + Directory Discovery)

feroxbuster -u http://target.com

feroxbuster -u http://target.com -x php,txt,html

feroxbuster -u http://target.com -r

OWASP ZAP Spider

zap.sh

Inside ZAP:

Target Site
→ Attack
→ Spider

Burp Suite Crawler