What is Nostr?
Raphael Lullis /
npub1e9d…zl6p
2024-06-13 16:11:05
in reply to nevent1q…vvke

Raphael Lullis on Nostr: robots.txt is no contract, no one should expect to be protected from being crawled ...

robots.txt is no contract, no one should expect to be protected from being crawled just because they placed some hints indicating what user agents are welcome.

Relying on consent and general good behavior is just bad digital hygiene. I'd rather people just acknowledged that and make it a basic principle: do not post anything on the open web unless you are sure you don't mind it being available to any present or future malicious/adversarial entity.
Author Public Key
npub1e9dq6selyjgrn4aludrq3drmmk9u9gflhtugtjgzmcqpku0zhtzqmdzl6p