Fabio Manganiello on Nostr: I’ve been working on #AI in some form or another for years, and I still don’t ...
I’ve been working on #AI in some form or another for years, and I still don’t think that it has anything that resembles actual intelligence - even with today’s most advanced models.
I define intelligence as the ability to extract general patterns out of data, even when incomplete, and be able to apply the same learnings even in very different scenarios that have apparently little in common with the original.
I only had to show my kid 2-3 cats before he learned to recognize any cats he sees. Even if they’re black rather than brown, or if one walks in the dark and he only saw cats during the day before, or if one of them is in a different position that he never saw before.
On the contrary, I have to train a neural network with thousands of pictures of cats, after appropriately ingesting them, labelling them and normalizing them, and carefully monitor the error rate during training to make sure that learning converges, in the hope that it can start telling the difference between a cat and a chair. Now imagine having to do it again and again for everything that you want your network to recognize.
The current state of the art of AI is still stuck in the statistical models space. Its aim is to calculate the coefficients of a large hyper-dimensional equation so that the most likely prediction that it spits out has the lowest probability of being wrong when compared to the data it’s been trained on. That’s it. Even the most sophisticated OpenAI models just apply minor variants to this principle, or transformations upstream/downstream - like, instead of predicting if there’s a cat or a chair in a photo, predict the next most likely word given this starting context, and after carefully modelling each word as a vector in a hyper-dimensional space, so you can start to better understand the nuances of human languages by comparing distances between vectors. Or build two networks, one that will be trained to produce an image, and another one trained to tell whether the image it produced was good.
And, of course, it adds billions of units and hundreds of layers to the network, so that the hyper-surface modelled by its equation fits as closely as possible as many possible inputs as possible. But that’s it. They are still, at all effects, statistical parrots. Maybe it doesn’t really matter for the end user - if it can solve my homework or generate an image for my blog, who cares if the result comes from brute-forced statistical models or from actual intelligence achieved through both deduction and inference? More sadly, it doesn’t necessarily have to be intelligent according to the philosophical definition in order to start replacing some human workers. If it quacks like a duck, then it must be a duck. And that’s unfortunate, because it means that currently the biggest companies invested into AI have very little incentives to risk new ways of doing things.
Things may change if AI started to properly incorporate logic predicates and semantics into its “knowledge base”. Curiously, that’s exactly what AI researchers used to do when I started working on it. My first job was in a company that used expert systems to translate content and extract context out of language, through large corpora of manually annotated linguistic definitions modelled as a graph - no neural networks involved. My first AI exam in college was all about first-order logic, decision trees and graph optimization problems. Neural networks were almost an appendix, mostly used for computer vision problems. When for fun I trained an AI in the year 2010 that could play tic-tac-toe using neural networks, some of my college folks raised an eyebrow - “why do you use neural networks to train an AI for a game where you could model the state of a game as a graph and use some heuristic to tell the next best move?”. It’s funny how the reaction is exactly the opposite in 2024 - “why did you bother to implement your own graph expansion algorithm and define your heuristic when you could just throw a neural network at the problem?”
At some point in the mid 2010s I believe that computing power started to become cheap and powerful enough however to make neural networks viable outside of academic contexts. So everybody started using these statistical artifacts to create “intelligence”, and the world of AI became a world where all problems are nails and all solutions are hammers. I still believe in a combination between logic and statistical models to achieve actual AI - just like humans have both innate empirical and rational components in the way they learn, and they can learn to recognize a cat without necessarily seeing thousands of them in a controlled environment.
https://www.newscientist.com/article/mg26335091-000-the-ai-expert-who-says-artificial-general-intelligence-is-nonsense/
I define intelligence as the ability to extract general patterns out of data, even when incomplete, and be able to apply the same learnings even in very different scenarios that have apparently little in common with the original.
I only had to show my kid 2-3 cats before he learned to recognize any cats he sees. Even if they’re black rather than brown, or if one walks in the dark and he only saw cats during the day before, or if one of them is in a different position that he never saw before.
On the contrary, I have to train a neural network with thousands of pictures of cats, after appropriately ingesting them, labelling them and normalizing them, and carefully monitor the error rate during training to make sure that learning converges, in the hope that it can start telling the difference between a cat and a chair. Now imagine having to do it again and again for everything that you want your network to recognize.
The current state of the art of AI is still stuck in the statistical models space. Its aim is to calculate the coefficients of a large hyper-dimensional equation so that the most likely prediction that it spits out has the lowest probability of being wrong when compared to the data it’s been trained on. That’s it. Even the most sophisticated OpenAI models just apply minor variants to this principle, or transformations upstream/downstream - like, instead of predicting if there’s a cat or a chair in a photo, predict the next most likely word given this starting context, and after carefully modelling each word as a vector in a hyper-dimensional space, so you can start to better understand the nuances of human languages by comparing distances between vectors. Or build two networks, one that will be trained to produce an image, and another one trained to tell whether the image it produced was good.
And, of course, it adds billions of units and hundreds of layers to the network, so that the hyper-surface modelled by its equation fits as closely as possible as many possible inputs as possible. But that’s it. They are still, at all effects, statistical parrots. Maybe it doesn’t really matter for the end user - if it can solve my homework or generate an image for my blog, who cares if the result comes from brute-forced statistical models or from actual intelligence achieved through both deduction and inference? More sadly, it doesn’t necessarily have to be intelligent according to the philosophical definition in order to start replacing some human workers. If it quacks like a duck, then it must be a duck. And that’s unfortunate, because it means that currently the biggest companies invested into AI have very little incentives to risk new ways of doing things.
Things may change if AI started to properly incorporate logic predicates and semantics into its “knowledge base”. Curiously, that’s exactly what AI researchers used to do when I started working on it. My first job was in a company that used expert systems to translate content and extract context out of language, through large corpora of manually annotated linguistic definitions modelled as a graph - no neural networks involved. My first AI exam in college was all about first-order logic, decision trees and graph optimization problems. Neural networks were almost an appendix, mostly used for computer vision problems. When for fun I trained an AI in the year 2010 that could play tic-tac-toe using neural networks, some of my college folks raised an eyebrow - “why do you use neural networks to train an AI for a game where you could model the state of a game as a graph and use some heuristic to tell the next best move?”. It’s funny how the reaction is exactly the opposite in 2024 - “why did you bother to implement your own graph expansion algorithm and define your heuristic when you could just throw a neural network at the problem?”
At some point in the mid 2010s I believe that computing power started to become cheap and powerful enough however to make neural networks viable outside of academic contexts. So everybody started using these statistical artifacts to create “intelligence”, and the world of AI became a world where all problems are nails and all solutions are hammers. I still believe in a combination between logic and statistical models to achieve actual AI - just like humans have both innate empirical and rational components in the way they learn, and they can learn to recognize a cat without necessarily seeing thousands of them in a controlled environment.
https://www.newscientist.com/article/mg26335091-000-the-ai-expert-who-says-artificial-general-intelligence-is-nonsense/