MarcusSchwemer on Nostr: To be honest: The robots.txt file is based on honesty. If a crawler does not respect ...

To be honest: The robots.txt file is based on honesty.

If a crawler does not respect your wishes, you are fucked up.
You can block a user agent in the htaccess file ... if it changes the the string randomly, you are fucked up.

Blocking AI crawlers is impossible ... they will eat everything they will get.