Eric the Cerise on Nostr: #AskFedi Okay ... I am working on developing a counter-culture #RobotsTxt ...
#AskFedi
Okay ... I am working on developing a counter-culture #RobotsTxt (robots.txt) file that excludes the major search engine crawlers ('cuz fuck them), but allows for assorted hobbyist, experimental, small-web-focused and other non-corporate crawlers ... I'm looking at search engines like https://wiby.me/ and https://search.marginalia.nu/ and https://stract.com/ and the like.
In the interests of not re-inventing the wheel, does anyone know of any good examples, or resources along these lines?
Thanks in advance
Okay ... I am working on developing a counter-culture #RobotsTxt (robots.txt) file that excludes the major search engine crawlers ('cuz fuck them), but allows for assorted hobbyist, experimental, small-web-focused and other non-corporate crawlers ... I'm looking at search engines like https://wiby.me/ and https://search.marginalia.nu/ and https://stract.com/ and the like.
In the interests of not re-inventing the wheel, does anyone know of any good examples, or resources along these lines?
Thanks in advance