John-Mark Gurney on Nostr: Does anyone have a nice little crawler trap bot? e.g. publish a robots.txt and ...

Does anyone have a nice little crawler trap bot?

e.g. publish a robots.txt and dynamically [or otherwise] generate random pages, and catch when crawlers ignoring your robots.txt so you can ban them?

I just blocked a couple hosts in 47.76/16 that were crawling my site, but didn't put who they were in their user-agent.