Neil Craig on Nostr: We recently noticed a fair bit of traffic on www.bbc.co.uk & www.bbc.com from a User ...

We recently noticed a fair bit of traffic on www.bbc.co.uk & www.bbc.com from a User Agent which identifies itself as "ByteSpider" (& has a @bytedance.com email address).

Lots of docs on the web state it doesn't obey robots.txt but ByteDance have told us it *does*:

> *...in the robots.txt files*
*> user-agent:Bytespider*
*> Disallow:/*

Thought that might be worth documenting as it might be a recent change & several of us searched but found zero docs from ByteDance

#SEO #ByteSpider #RobotsTXT