What is Nostr?
Simon Willison /
npub13v9…w5eu
2023-06-08 20:38:57

Simon Willison on Nostr: Understanding GPT tokenizers: I wrote about how the tokenizers used by the various ...

Understanding GPT tokenizers: I wrote about how the tokenizers used by the various GPT models actually work, including an interactive tool for experimenting with their output https://simonwillison.net/2023/Jun/8/gpt-tokenizers/
Author Public Key
npub13v97j0kknscwnf5pt87nsn7cxzxwfwl3dsu7ss8qsq7ukmqgwg8q84w5eu