petersuber on Nostr: "AI crawler wars threaten to make the web more closed for everyone" ...
"AI crawler wars threaten to make the web more closed for everyone"
https://www.technologyreview.com/2025/02/11/1111518/ai-crawler-wars-closed-web/
(#paywalled)
"Crawlers are endemic. Now representing half of all internet traffic, they will soon outpace human traffic… As of late, they serve one more purpose: Companies such as OpenAI use web-crawled data to train their artificial intelligence systems…Websites are now fighting back for fear that this invasive species…will help displace them. But…this pushback is also threatening the transparency and open borders of the web…Unless we are thoughtful about how we fix this, the web will increasingly be fortified with logins, paywalls, and access tolls that inhibit not just AI but the biodiversity of real users and useful crawlers."
#AI #LLMs #OpenAccess #Risks
https://www.technologyreview.com/2025/02/11/1111518/ai-crawler-wars-closed-web/
(#paywalled)
"Crawlers are endemic. Now representing half of all internet traffic, they will soon outpace human traffic… As of late, they serve one more purpose: Companies such as OpenAI use web-crawled data to train their artificial intelligence systems…Websites are now fighting back for fear that this invasive species…will help displace them. But…this pushback is also threatening the transparency and open borders of the web…Unless we are thoughtful about how we fix this, the web will increasingly be fortified with logins, paywalls, and access tolls that inhibit not just AI but the biodiversity of real users and useful crawlers."
#AI #LLMs #OpenAccess #Risks