The maths you need to start understanding LLMs (gilesthomas.com)

Article summary

The article explains the mathematical concepts necessary to understand Large Language Models (LLMs), specifically vectors, high-dimensional spaces, and embeddings. It discusses how these concepts are used to represent likelihoods and meanings in LLMs. The article aims to provide a foundation for understanding LLMs, but notes that it only covers the basics and that true understanding requires more in-depth knowledge. The author also distinguishes between the 'messy' unnormalised space and the neat, tidy normalised one, where probabilities are used to represent likelihoods.

Main themes

• Mathematics for LLMs
• Vector spaces
• Embeddings
• LLM understanding
• High-dimensional spaces
• Probability distributions

What commenters say

• Understanding LLMs requires a foundation in mathematical concepts such as linear algebra and probability, but having a formula is not the same as true understanding.
• The article provides a necessary but not sufficient introduction to the math behind LLMs, and more knowledge is needed to fully comprehend them.
• LLMs can be viewed as simply predicting the next token in a sequence, but this perspective is inadequate for understanding their true capabilities and limitations.
• The distinction between predicting words statistically and modeling the world is a crucial one, with some arguing that LLMs do more than just predict words.
• The complexity of LLMs and their emergent properties make them difficult to fully understand, even for those with a strong foundation in the underlying math.
• The article's explanation of math for LLMs is relevant for those who want to understand how they work, but not necessary for the general public who just want to use them.
• The normalization of cloud-based processing and surveillance infrastructure is a concern that goes beyond the technical details of LLMs and into the realm of societal implications.
• The question of whether LLMs truly 'understand' the world or are just complex statistical models is a topic of debate, with some arguing that they are simply approximating a mathematical function.