david_chisnall on Nostr: I wonder if the lesson from the current hype wave is that Yahoo! was right all along. ...
I wonder if the lesson from the current hype wave is that Yahoo! was right all along.
Back in the ‘90s, Yahoo! was a search engine that worked by having a human-curated list of sites that you could search. They gradually lost out to AltaVista (originally created as a tech demo to show how fast the DEC Alpha was), which mechanically crawled the web and indexed things. AltaVista eventually lost out to Google, which loaded faster (because the founders weren’t good at HTML and so created a very simple page) and which used PageRank to find sites that lots of people linked to and give them a higher score, so used a distributed reputation system (which later proved to be easily gamed and so ended up being replaced with other mechanisms).
Yahoo! lost because their employees could index things far more slowly than people could write things on the web. This would matter far less in a distributed setting because humans reading web content can read it far faster than other humans can write it. On the other hand, LLMs can produce slop far faster than humans can read it. The proportion of web content that is worth indexing is rapidly dropping.
Back in the ‘90s, Yahoo! was a search engine that worked by having a human-curated list of sites that you could search. They gradually lost out to AltaVista (originally created as a tech demo to show how fast the DEC Alpha was), which mechanically crawled the web and indexed things. AltaVista eventually lost out to Google, which loaded faster (because the founders weren’t good at HTML and so created a very simple page) and which used PageRank to find sites that lots of people linked to and give them a higher score, so used a distributed reputation system (which later proved to be easily gamed and so ended up being replaced with other mechanisms).
Yahoo! lost because their employees could index things far more slowly than people could write things on the web. This would matter far less in a distributed setting because humans reading web content can read it far faster than other humans can write it. On the other hand, LLMs can produce slop far faster than humans can read it. The proportion of web content that is worth indexing is rapidly dropping.