Simon Willison on Nostr: Understanding GPT tokenizers: I wrote about how the tokenizers used by the various ...
Published at
2023-06-08 20:38:57Event JSON
{
"id": "625798861d48b5df0fa62dad302fa8e2b834a24572a9d3c93933a654b13593c4",
"pubkey": "8b0be93ed69c30e9a68159fd384fd8308ce4bbf16c39e840e0803dcb6c08720e",
"created_at": 1686256737,
"kind": 1,
"tags": [
[
"mostr",
"https://fedi.simonwillison.net/users/simon/statuses/110510521550785911"
]
],
"content": "Understanding GPT tokenizers: I wrote about how the tokenizers used by the various GPT models actually work, including an interactive tool for experimenting with their output https://simonwillison.net/2023/Jun/8/gpt-tokenizers/",
"sig": "ee9d696600efa53cc801ded4b0df237922b113b8c87c405cbd4580129a000747d7d5a52a37b232ddd32c288e88dec49759c24cda8bae8fbc151243c35c935f2d"
}
