Marcel Bischoff on Nostr: Anyone else seeing that an #OpenAI crawler is ignoring robots.txt directives? The ...
Anyone else seeing that an #OpenAI crawler is ignoring robots.txt directives? The user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
doesn't even request a robots.txt file but attempts to slowly catalog my entire website. I have blocked this user agent now. How is this acceptable behaviour?
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
doesn't even request a robots.txt file but attempts to slowly catalog my entire website. I have blocked this user agent now. How is this acceptable behaviour?