Stefan Bohacek on Nostr: Just throwing out a thought before I do some research on this, but I think robots.txt ...

Just throwing out a thought before I do some research on this, but I think robots.txt needs an update.

Ideally I'd like to define an "allow list" that tells web scrapers how my content can be used. Eg.:

- monetizable: false
- fediverse: true
- nonfediverse: false
- ai: false

Etc. And I'd like to apply this to my social media profile and any other web presence, not just my personal website.

#internet #fediverse #SocialMedia #robotsTxt