Jan Beta on Nostr: Is there a list of things to add to a robots.txt file to prevent AI scraping and ...
Is there a list of things to add to a robots.txt file to prevent AI scraping and such? (Or at least tell the scrapers to F off, I am well aware they mostly don’t care.)
Published at
2025-01-15 13:50:07Event JSON
{
"id": "63780f32430cf200d66ba3de405b72c12a04d1c21be4b7e96cae5d784da61a52",
"pubkey": "ac92102a2ecb873c488e0125354ef5a97075a16198668c360eda050007ed42cd",
"created_at": 1736949007,
"kind": 1,
"tags": [
[
"proxy",
"https://chaos.social/users/janbeta/statuses/113832690134004918",
"activitypub"
]
],
"content": "Is there a list of things to add to a robots.txt file to prevent AI scraping and such? (Or at least tell the scrapers to F off, I am well aware they mostly don’t care.)",
"sig": "d5c94905953adffd4acbaad9237c84ed2c14cc8712596957b606c4335952cdbe0058d2a4c951302f6dcd821062956f1b469753d10065224385d7221add1c7694"
}
