Dad on Nostr: Some bots to put in robots.txt User-agent: AdsBot-Google Disallow: / User-agent: ...
Some bots to put in robots.txt
User-agent: AdsBot-Google
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: panscient.com
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Turnitin
Disallow: /
User-agent: AdsBot-Google
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: panscient.com
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Turnitin
Disallow: /