Marc W Howard on Nostr: npub1skvad…laky3 So for #2 the Bellman equation solves a problem---estimate ...
npub1skvad2l2wrxgdmt6yxk9kt2rjhw5tucjzhf54pktfq2gg0qhgwyqdlaky3 (npub1skv…aky3)
So for #2 the Bellman equation solves a problem---estimate expected future reward without directly estimating the future---that is not faced by the brain. Insofar as the brain has an explicit temporal memory of the continuous past (and it certainly does), it is straightforward to construct a direct estimate of the distant future via simple Hebbian associations. We didn't know that in the 80's when Sutton and Barto were working on this problem or the mid-90's when the dopamine mapping was made. This paper makes these arguments in much more depth:
https://arxiv.org/abs/2302.10163
(will try to write answer to #3 later, time for dinner)
So for #2 the Bellman equation solves a problem---estimate expected future reward without directly estimating the future---that is not faced by the brain. Insofar as the brain has an explicit temporal memory of the continuous past (and it certainly does), it is straightforward to construct a direct estimate of the distant future via simple Hebbian associations. We didn't know that in the 80's when Sutton and Barto were working on this problem or the mid-90's when the dopamine mapping was made. This paper makes these arguments in much more depth:
https://arxiv.org/abs/2302.10163
(will try to write answer to #3 later, time for dinner)