Fabio Manganiello on Nostr: Almost half of the online news publishers have blocked OpenAI and similar AI engines ...
Almost half of the online news publishers have blocked OpenAI and similar AI engines from scanning their pages via robots.txt.
Thus, in a short time we will have:
1. News publishers that don't block AI crawlers. All of their content will be sucked up by the bots and repackaged into conversational answers. These publishers will struggle more and more to monetize their content, as users won't have many incentives to pay for a NYT subscription when they can just ask questions to ChatGPT and get a summary of any article. As their monetization struggle goes on, the quality of their content will go down - less money means less spending on qualified staff, and their content will turn more and more into clickbait garbage in order to maximize the click-through rate.
2. News publishers that block AI crawlers. You may soon be unable to find some of those articles even if you type the right keywords into a search engine. They'll try to be their own independent portals, and the only way for you to discover their content will be to type their URL in your browser, like it's 1997. They're likely to make their paywalls even more aggressive, and they'll probably make sure that not a single line of text leaks to non-authenticated connections, thus hindering our ability to index and preserve the web. It's like walking into a supermarket, and finding out that all the newspapers have been covered by a blanket, and even reading the headlines requires you to purchase a yearly subscription.
Congratulations. Tech has managed to fuck up the news industry for good, and bring it to a state where it's even less accessible than it used to be 20 years ago.
https://flowingdata.com/2023/10/09/news-organizations-blocking-openai/
Thus, in a short time we will have:
1. News publishers that don't block AI crawlers. All of their content will be sucked up by the bots and repackaged into conversational answers. These publishers will struggle more and more to monetize their content, as users won't have many incentives to pay for a NYT subscription when they can just ask questions to ChatGPT and get a summary of any article. As their monetization struggle goes on, the quality of their content will go down - less money means less spending on qualified staff, and their content will turn more and more into clickbait garbage in order to maximize the click-through rate.
2. News publishers that block AI crawlers. You may soon be unable to find some of those articles even if you type the right keywords into a search engine. They'll try to be their own independent portals, and the only way for you to discover their content will be to type their URL in your browser, like it's 1997. They're likely to make their paywalls even more aggressive, and they'll probably make sure that not a single line of text leaks to non-authenticated connections, thus hindering our ability to index and preserve the web. It's like walking into a supermarket, and finding out that all the newspapers have been covered by a blanket, and even reading the headlines requires you to purchase a yearly subscription.
Congratulations. Tech has managed to fuck up the news industry for good, and bring it to a state where it's even less accessible than it used to be 20 years ago.
https://flowingdata.com/2023/10/09/news-organizations-blocking-openai/