someone on Nostr: GRPO & EGO GRPO is a training algorithm introduced by R1. Why is it a big deal? It ...
GRPO & EGO
GRPO is a training algorithm introduced by R1. Why is it a big deal? It allowed models to reject themselves.
A model outputs some words while trying to solve a math or coding problem. If it cannot solve, the next round it may try a longer reasoning. And while doing all of this at some point "Wait!" or "But," or "On the other hand" is randomly introduced in the reasoning words and that allows it to re-think its reasoning words and correct itself. Once these random appearances of reflection allows it to solve problems, the next round it wants to do more of that because it got rewards when it did that. Hence it gets smarter gradually thanks to self reflection.
I think this is better than SFT because it fixes its own errors while SFT is primarily focusing on teaching new skills. Inverting the error is kind of "fixing the mistakes in itself" (GRPO method) and could be more effective than installing new ideas and hoping old ideas go away (SFT method).
LLMs fixing their own errors allows them to self learn. This has analogies to human operation. Rejecting the ego is liberation from the shackles of ego, in this case the past words are kind of shackles but when it corrects itself it is "thinking outside the box". We find our mistakes and contemplate on them and learn from them and next time don't repeat. We f around and find out basically. F around is enjoying life recklessly, finding out is "divine scripts work most of the time and should have priority in decision making". Controlling the ego and getting outside of the box of ego is how we ascend.
GRPO is a training algorithm introduced by R1. Why is it a big deal? It allowed models to reject themselves.
A model outputs some words while trying to solve a math or coding problem. If it cannot solve, the next round it may try a longer reasoning. And while doing all of this at some point "Wait!" or "But," or "On the other hand" is randomly introduced in the reasoning words and that allows it to re-think its reasoning words and correct itself. Once these random appearances of reflection allows it to solve problems, the next round it wants to do more of that because it got rewards when it did that. Hence it gets smarter gradually thanks to self reflection.
I think this is better than SFT because it fixes its own errors while SFT is primarily focusing on teaching new skills. Inverting the error is kind of "fixing the mistakes in itself" (GRPO method) and could be more effective than installing new ideas and hoping old ideas go away (SFT method).
LLMs fixing their own errors allows them to self learn. This has analogies to human operation. Rejecting the ego is liberation from the shackles of ego, in this case the past words are kind of shackles but when it corrects itself it is "thinking outside the box". We find our mistakes and contemplate on them and learn from them and next time don't repeat. We f around and find out basically. F around is enjoying life recklessly, finding out is "divine scripts work most of the time and should have priority in decision making". Controlling the ego and getting outside of the box of ego is how we ascend.