sandwich on Nostr: nostr.watch daemons can soon be blocked via `robots.txt` Blocking nostr.watch daemons ...
nostr.watch daemons can soon be blocked via `robots.txt`
Blocking nostr.watch daemons will eventually result in an Uptime of 0% on the site, limited data availability on nostr.watch and will exclude your relay's data from global historical data, which has not yet been revealed.
Robots.txt will not affect-clientside checks, and your relay will still be listed on nostr.watch. Delistings of online relays is not currently supported.
If your relay is `wss://relay.com` then the robots.txt location would be `https://relay.com/robots.txt`
Using robots.txt is not exactly standard, but was easy to piggy-back on, it is temporary. The better solution would be an amendment to NIP-11 of some sort. Robots.txt parsing by nostr.watch daemons will be deprecated when there is a suitable alternative.
If your robots.txt is currently disallowing all User-agents, but you wish to allow nostr.watch, add:
User-agent: nostr.watch
Allow: /
Please be aware the daemons are getting more performant, optimized and polite with each passing day, they were pretty rude out of the gate, largely due to a feedback loop between two disparate bugs. Sorry about that.
Blocking nostr.watch daemons will eventually result in an Uptime of 0% on the site, limited data availability on nostr.watch and will exclude your relay's data from global historical data, which has not yet been revealed.
Robots.txt will not affect-clientside checks, and your relay will still be listed on nostr.watch. Delistings of online relays is not currently supported.
If your relay is `wss://relay.com` then the robots.txt location would be `https://relay.com/robots.txt`
Using robots.txt is not exactly standard, but was easy to piggy-back on, it is temporary. The better solution would be an amendment to NIP-11 of some sort. Robots.txt parsing by nostr.watch daemons will be deprecated when there is a suitable alternative.
If your robots.txt is currently disallowing all User-agents, but you wish to allow nostr.watch, add:
User-agent: nostr.watch
Allow: /
Please be aware the daemons are getting more performant, optimized and polite with each passing day, they were pretty rude out of the gate, largely due to a feedback loop between two disparate bugs. Sorry about that.