Dan Goodman on Nostr: npub1s4kpx…8j02h npub15swlx…zx855 this is a great question. I've been thinking ...
npub1s4kpxq9cwpx6g3tdmrk7fd8eeatncjmw4wqzszs6s3mvmrwl67qqc8j02h (npub1s4k…j02h) npub15swlxudlhx4ttcgsd4556zuqrl57qndxmt4n3dnzrkqn89nxv6lsjzx855 (npub15sw…x855) this is a great question. I've been thinking about it all evening and my answer surprised myself.
When you read a really good and careful experimental paper, and you go through the methods in detail, there's just this staggering attention to detail and willingness to consider all the ways they could be accidentally tricking themselves into seeing what they want to see.
Like, they recorded responses to two different types of stimulus to see the different responses, but then they realised that the ancient PC they were using to generate the stimuli had different levels of fan noise depending which stimuli it was and the PC was in the chamber, so they put the chamber in a sound proof booth. But they didn't stop there, they got high precision calibration equipment to measure the received noise in the booth and it was just still just detectable. So they put the booth on stilts to minimise the vibrations. Then they couldn't detect the noise but they still didn't trust it wasn't there so they filled the room outside with a bunch of PCs generating the same types of stimuli on random uncorrelated schedules to mask the true signal, and so on.
As a computational person it's shameful to say it, but even though this level of attention to detail would be so much easier for us, we just don't do it. I think we're a little bit too willing to allow ourselves to be tricked by our models or simulations and don't put the effort in to stopping it from happening. Our methods sections are just recipes with no explanation as to why 5 layers were used instead of 4, etc.
And to get back to the specifics we were talking about, it's so much easier to trick yourself with a massively complex model with millions of parameters that's so computationally expensive to run that it can only be run a handful of times, and a measure that is so involved we can only guess at what it's actually measuring.
When you read a really good and careful experimental paper, and you go through the methods in detail, there's just this staggering attention to detail and willingness to consider all the ways they could be accidentally tricking themselves into seeing what they want to see.
Like, they recorded responses to two different types of stimulus to see the different responses, but then they realised that the ancient PC they were using to generate the stimuli had different levels of fan noise depending which stimuli it was and the PC was in the chamber, so they put the chamber in a sound proof booth. But they didn't stop there, they got high precision calibration equipment to measure the received noise in the booth and it was just still just detectable. So they put the booth on stilts to minimise the vibrations. Then they couldn't detect the noise but they still didn't trust it wasn't there so they filled the room outside with a bunch of PCs generating the same types of stimuli on random uncorrelated schedules to mask the true signal, and so on.
As a computational person it's shameful to say it, but even though this level of attention to detail would be so much easier for us, we just don't do it. I think we're a little bit too willing to allow ourselves to be tricked by our models or simulations and don't put the effort in to stopping it from happening. Our methods sections are just recipes with no explanation as to why 5 layers were used instead of 4, etc.
And to get back to the specifics we were talking about, it's so much easier to trick yourself with a massively complex model with millions of parameters that's so computationally expensive to run that it can only be run a handful of times, and a measure that is so involved we can only guess at what it's actually measuring.