laanwj on Nostr: it reminds me of the kind of tokenization that LLMs do (though that isn't strictly ...
it reminds me of the kind of tokenization that LLMs do (though that isn't strictly per word)
Published at
2024-11-20 20:06:32Event JSON
{
"id": "73b44aaaad35740f965e7a72e64ac091c0c53c038369d2f2ad75fc6fe34d2e74",
"pubkey": "0aa39e5aef99a000a7bdb0b499158c92bc4aa20fb65931a52d055b5eb6dff738",
"created_at": 1732133192,
"kind": 1,
"tags": [
[
"e",
"70fc146a8bbde7628cef73e54e96455eba0e05ba567ef491bf08ead9cd547700",
"wss://nos.lol/",
"root"
],
[
"e",
"29d740e651c5bd2172cd7e1f15019354588d2258a9fa96c74563b0c0ee16b82e",
"wss://nos.lol/",
"reply"
],
[
"p",
"32e1827635450ebb3c5a7d12c1f8e7b2b514439ac10a67eef3d9fd9c5c68e245",
"",
"mention"
],
[
"p",
"460c25e682fda7832b52d1f22d3d22b3176d972f60dcdc3212ed8c92ef85065c",
"",
"mention"
]
],
"content": "it reminds me of the kind of tokenization that LLMs do (though that isn't strictly per word)",
"sig": "9904fa6b5efce0d45e65525e2a06b96332c9c4b47df7674891bc24aabee123419b5188b4205e2b7c9ef65da525c3b7c869262fdf1a8d3303e03d24afed72695e"
}
