Deus on Nostr: So I read somewhere that just like disallow all in Robots.txt works if the bots ...
So I read somewhere that just like disallow all in Robots.txt works if the bots respect it or not. And that it's the same case with these "AI scrapers" and there's no way to 'brute force' stop them, am I right? Has anyone arrived at a solution to this?
Published at
2024-09-27 16:01:48Event JSON
{
"id": "81dd79fda8a695306be2a3a191bbd49aa287fe987b8ec97a5a494e9d20c953c2",
"pubkey": "b60f85fcb8c13a0b7c2e17403ffd8ba49ed0b48dd63ff58009d342520ad8b922",
"created_at": 1727452908,
"kind": 1,
"tags": [
[
"p",
"eb4c37301743a8655dc6c7607ea6d3af3449f47a9ff7ae1104588843740a5462",
"wss://relay.mostr.pub"
],
[
"e",
"560c12fe03628e8900a8f28b8b5eaee81b3f8619663c4059d05df1c3ebfdc8f0",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://charcha.cc/objects/a05f7206-e279-42ed-977b-2215ae384655",
"activitypub"
]
],
"content": "So I read somewhere that just like disallow all in Robots.txt works if the bots respect it or not. And that it's the same case with these \"AI scrapers\" and there's no way to 'brute force' stop them, am I right? Has anyone arrived at a solution to this?",
"sig": "32f358f771f5250d176482e4f0493f3ffdd97e44f28fc7c0c33e7b1f6df0d6acf5efa6d6874db40a3edaaf51a9cc6a026d5100bc2407116901422e994366f6af"
}
