The last six months in LLMs, illustrated by pelicans on bicycles (simonwillison.net)

Article summary

The article discusses the recent developments in Large Language Models (LLMs) over the last six months, highlighting notable model releases and their capabilities. The author uses a unique benchmark, asking LLMs to generate an SVG of a pelican riding a bicycle, to evaluate their performance. This test is considered unreasonably difficult, as it requires the models to generate code and understand the concept of a pelican and a bicycle. The author also touches on the issue of bugs in LLMs and the importance of understanding their limitations.

Main themes

• LLM development
• benchmarking
• probabilistic models
• human vs LLM capabilities
• marketing and expectations
• quality and improvement
• cost and speed tradeoffs

What commenters say

• The pelican on a bicycle test is an inappropriate use case for text-only models, as it requires a level of visual understanding that they are not designed for.
• The test is a useful way to evaluate the baseline smartness of each model and can be a predictor of performance on other tasks.
• The quality of the generated images is not the main point of the test, but rather the ability of the models to follow a spec and generate code.
• The comparison of LLMs using a single sample is not sufficient, as probabilistic models should be evaluated over multiple samples.
• The marketing of LLMs as discrete, human-like models is misleading, as they are inherently probabilistic and may not always produce perfect results.
• The ability of humans to judge their performance and improve over time is a key advantage over LLMs, which may produce low-quality results quickly but struggle to improve.
• The cost and speed advantage of LLMs is significant, but comes at the cost of extremely low quality, and may not be suitable for all use cases.