What is Nostr?
tom jennings /
npub1qd2…74he
2023-11-26 00:06:32

tom jennings on Nostr: Claude Shannon explored a simpler version of the ideas that LLMs use, in his 1948 ...

Claude Shannon explored a simpler version of the ideas that LLMs use, in his 1948 paper, "A mathemetical theory of information". The paper is a mix of arcane (to me) math and nice prose.

He plays with known letter and word frequency and redundancy to create strings that appear to contain information.

Starting here (page 4):
2. THE DISCRETE SOURCE OF INFORMATION

or if you want a quicker look (page 6):

3. THE SERIES OF APPROXIMATIONS TO ENGLISH

https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
Author Public Key
npub1qd2pnra30h4mnslpdt66rweeqwnvgc00l4rv7tz0ge99zs2ypfeqw674he