Nicolas Cropp on Nostr: robots.txt is just a suggestion and nothing is stopping the bots from ignoring it on ...
robots.txt is just a suggestion and nothing is stopping the bots from ignoring it on “accident”.
We need a better, enforceable solution to prevent AI from gobbling up the web.
#ai #robotstxt #crawler #privacy #openai #perplexity
Published at
2024-06-26 15:43:16Event JSON
{
"id": "6623ec442ba6a08734352307b96280494819d02b9818adfbac23485eedd3b111",
"pubkey": "635c07efd8da0d6548579697ee54d2f0a2b436b9987a9593fa81d632264a605d",
"created_at": 1719416596,
"kind": 1,
"tags": [
[
"t",
"ai"
],
[
"t",
"robotstxt"
],
[
"t",
"crawler"
],
[
"t",
"privacy"
],
[
"t",
"openai"
],
[
"t",
"perplexity"
],
[
"proxy",
"https://social.lol/users/nicolas/statuses/112683686065856474",
"activitypub"
]
],
"content": "robots.txt is just a suggestion and nothing is stopping the bots from ignoring it on “accident”. \n\nWe need a better, enforceable solution to prevent AI from gobbling up the web. \n\n#ai #robotstxt #crawler #privacy #openai #perplexity\n\nhttps://media.social.lol/media_attachments/files/112/683/685/843/154/175/original/8d6771c6cd8fba26.jpeg",
"sig": "ffe9e64e30a6c13a7a0b13100eba310128b5dc6cfa2e6507f606f5090894af9f9a40742cb43c31690484d974ef3e9664267f82c52a51dbb09879abac048d11c4"
}
