tom jennings on Nostr: Claude Shannon explored a simpler version of the ideas that LLMs use, in his 1948 ...
Claude Shannon explored a simpler version of the ideas that LLMs use, in his 1948 paper, "A mathemetical theory of information". The paper is a mix of arcane (to me) math and nice prose.
He plays with known letter and word frequency and redundancy to create strings that appear to contain information.
Starting here (page 4):
2. THE DISCRETE SOURCE OF INFORMATION
or if you want a quicker look (page 6):
3. THE SERIES OF APPROXIMATIONS TO ENGLISH
https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
He plays with known letter and word frequency and redundancy to create strings that appear to contain information.
Starting here (page 4):
2. THE DISCRETE SOURCE OF INFORMATION
or if you want a quicker look (page 6):
3. THE SERIES OF APPROXIMATIONS TO ENGLISH
https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf