An LLM is a lossy encyclopedia (simonwillison.net)

Article summary

The article compares a Large Language Model (LLM) to a lossy encyclopedia, suggesting that while it can store a vast amount of information, the compression is lossy and may not always provide accurate or detailed answers. The author argues that it's essential to develop an intuition for what questions an LLM can usefully answer and what questions require more specific knowledge. The article also discusses the limitations of LLMs in providing extremely specific facts, such as creating a boilerplate project skeleton for a specific hardware configuration. The author suggests treating LLMs as tools that can act on presented facts rather than expecting them to know everything.

Main themes

• LLM limitations
• Lossy compression
• Knowledge retrieval
• Language understanding
• AI expectations
• Human-AI interaction

What commenters say

• LLMs are not oracles and should not be expected to provide perfect answers, but rather used as tools to assist in finding information.
• The value of LLMs lies in their ability to provide quick answers and unify interfaces, rather than replacing human knowledge or expertise.
• The limitations of LLMs are not fundamental, but rather a result of training methods and data, which can be improved over time.
• LLMs can be seen as lossy encyclopedias, but also as clever search engines or translator engines, depending on the context and application.
• The analogy of an LLM as a lossy encyclopedia is useful, but may not capture all the capabilities and limitations of these models.
• Overconfidence in LLMs can lead to incorrect decisions and outcomes, and it's essential to critically evaluate their output.
• The ability of LLMs to ask clarifying questions is seen as both a benefit and a drawback, depending on the user's goals and expectations.
• LLMs may become snapshots of human knowledge and societal biases, which can be useful for future historians and researchers.