Stefan Bohacek on Nostr: Just throwing out a thought before I do some research on this, but I think robots.txt ...
Just throwing out a thought before I do some research on this, but I think robots.txt needs an update.
Ideally I'd like to define an "allow list" that tells web scrapers how my content can be used. Eg.:
- monetizable: false
- fediverse: true
- nonfediverse: false
- ai: false
Etc. And I'd like to apply this to my social media profile and any other web presence, not just my personal website.
#internet #fediverse #SocialMedia #robotsTxt
Published at
2024-06-12 15:27:44Event JSON
{
"id": "d67238870ffb478bfbff8510c5f542a8a4ca45af2a56e6c8eb36bcecfb0020d1",
"pubkey": "b031dd8fe41796c0aa868e18b7094ba180c6974ecc414ccffa527e979ff6ac94",
"created_at": 1718206064,
"kind": 1,
"tags": [
[
"t",
"socialmedia"
],
[
"t",
"robotstxt"
],
[
"proxy",
"https://stefanbohacek.online/@stefan/112604352640135688",
"web"
],
[
"t",
"fediverse"
],
[
"t",
"internet"
],
[
"proxy",
"https://stefanbohacek.online/users/stefan/statuses/112604352640135688",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://stefanbohacek.online/users/stefan/statuses/112604352640135688",
"pink.momostr"
]
],
"content": "Just throwing out a thought before I do some research on this, but I think robots.txt needs an update. \n\nIdeally I'd like to define an \"allow list\" that tells web scrapers how my content can be used. Eg.:\n\n- monetizable: false\n- fediverse: true\n- nonfediverse: false\n- ai: false\n\nEtc. And I'd like to apply this to my social media profile and any other web presence, not just my personal website.\n\n#internet #fediverse #SocialMedia #robotsTxt",
"sig": "06ea64c34922bca2b3d191e30257798d79c658a4f6fc1181db37cabc30c5a777e47f5d0b136778ebc1f1a450e504cd55fea0f43a5d78157f73858e45fab417b2"
}
