petri on Nostr: The unwritten rules of respecting robots.txt and not scooping up everything have ...

The unwritten rules of respecting robots.txt and not scooping up everything have already crumpled due to AI scraping.

https://www.businessinsider.com/meta-web-crawler-bots-robots-txt-ai-2024-8

Seems to be the trend that if you can access it it will be used. Encrypt and deny access might become the new reality, unfortunately.