Karthik Srinivasan on Nostr: A somewhat contrived question. I want to know if we can generalize or extend the idea ...
A somewhat contrived question. I want to know if we can generalize or extend the idea of model complexity used in machine learning, statistical learning theory, and deep nets. Alternately, what are its limits? (Indirectly, I want to tease apart what either “model” or “complexity” means in this worldview. They do not seem to be “models” that one uses in science typically nor their measure of complexity seem to line up with any scientific model).
In ML, we typically use, “the number of free parameters” as proxy for model complexity, i.e., weights and biases etc., then penalize for over-parameterization with some information criteria (AIC, BIC etc., etc.,), or regularize (with dropout etc.,) to arrive at bias-variance tradeoff curves. Then there is the double descent (Belkin et al.,) for deep-nets that seems to buck the trend of standard ML (kernel) methods.
My question: if we are to turn this approach to scientific models, how can we evaluate their complexity in the same machine learning framework (yes, I know this sounds bizarre)?
For example, take any simple physics model, as explanatory as it is, it is also predictive. If I were to treat it as a purely predictive model, and try to plot the bias-variance curve, what would constitute its complexity (x-axis), and where/when does its error start going up (y-axis, so the model breaks)? If we can’t do this ML-ification of a simple physics model, why not?
Corollary: Take some nonlinear dynamical system that is the model of some physical phenomenon, how do you define its model complexity? (Are the three parameters of a Lorentz attractor the model complexity of the system in ML parlance?)
#MachineLearning #Science #Models #ModelComplexity #BiasVariance
In ML, we typically use, “the number of free parameters” as proxy for model complexity, i.e., weights and biases etc., then penalize for over-parameterization with some information criteria (AIC, BIC etc., etc.,), or regularize (with dropout etc.,) to arrive at bias-variance tradeoff curves. Then there is the double descent (Belkin et al.,) for deep-nets that seems to buck the trend of standard ML (kernel) methods.
My question: if we are to turn this approach to scientific models, how can we evaluate their complexity in the same machine learning framework (yes, I know this sounds bizarre)?
For example, take any simple physics model, as explanatory as it is, it is also predictive. If I were to treat it as a purely predictive model, and try to plot the bias-variance curve, what would constitute its complexity (x-axis), and where/when does its error start going up (y-axis, so the model breaks)? If we can’t do this ML-ification of a simple physics model, why not?
Corollary: Take some nonlinear dynamical system that is the model of some physical phenomenon, how do you define its model complexity? (Are the three parameters of a Lorentz attractor the model complexity of the system in ML parlance?)
#MachineLearning #Science #Models #ModelComplexity #BiasVariance