Pavel Korytov :emacs:☮️ on Nostr: Relying on robots.txt to scare away crawlers is like using a 'no trespassing' sign ...
Relying on robots.txt to scare away crawlers is like using a 'no trespassing' sign instead of a lock. No legislation will change that as long as there's competitive advantage in ignoring the file. If you're serious about preventing parsing, use CAPTCHAs.
Or, if you're serious about your information being used (and about not annoying me when I try to contribute to #rssbridge), let it go.
Published at
2025-01-08 14:21:23Event JSON
{
"id": "acd655de08a15245ac102b774ac787f91fdb285dfb425d3ee3d5e1171cea392d",
"pubkey": "a0beb2d46e7c459f153950cf3af3c6e29ba1d316d08d9e92c9317c0687c99b55",
"created_at": 1736346083,
"kind": 1,
"tags": [
[
"t",
"rssbridge"
],
[
"proxy",
"https://mastodon.bsd.cafe/users/sqrtminusone/statuses/113793176907584774",
"activitypub"
]
],
"content": "Relying on robots.txt to scare away crawlers is like using a 'no trespassing' sign instead of a lock. No legislation will change that as long as there's competitive advantage in ignoring the file. If you're serious about preventing parsing, use CAPTCHAs.\n\nOr, if you're serious about your information being used (and about not annoying me when I try to contribute to #rssbridge), let it go.",
"sig": "54bae0a5202833c5b507b997311d167b46e5f321f311137dc803d7e466816e9f34fe93ed6f13cf3a6f6d8e4c9be4327876e13c1f9287e1fb4686bd8b31c344a7"
}
