Dare Obasanjo on Nostr: Wired has discovered that Perplexity’s crawler ignores robots.txt directives that ...

Wired has discovered that Perplexity’s crawler ignores robots.txt directives that block AI crawlers.

Given the choice of spending millions licensing content like Google & OpenAI or having outdated info, it’s unsurprising some startups are taking the third option.

Unethical, yes. Surprising, no.

https://www.wired.com/story/perplexity-is-a-bullshit-machine/