LLM Architecture Gallery (sebastianraschka.com)

Article summary

The LLM Architecture Gallery is a curated reference for modern language-model architectures, providing a comprehensive overview of 85 models. The gallery includes architecture diagrams, implementation links, and release dates, allowing for comparisons between different models. The gallery focuses on text-only LLMs and language-model backbones, with corrections and updates welcome through an issue tracker. The project is also available as a physical poster and digital download.

Main themes

• LLM Architecture Comparison
• Model Evolution
• Innovation and Progress
• Efficiency and Scalability
• Digital Evolution
• Human vs Machine Design
• Competitiveness and Fine-Tuning

What commenters say

• Some commenters appreciate the resource and suggest potential improvements, such as adding a sort order or scaled view to better understand the evolution of models.
• There is debate about whether there have been fundamental innovations in LLM architecture in recent years, with some arguing that improvements have been mainly incremental.
• The importance of innovations like MoE, GQA, and RoPE is discussed, with some considering them significant advancements.
• Others argue that competitiveness in LLMs comes from scale, data, and fine-tuning rather than architecture.
• The potential for digital evolution in LLMs is noted, with some drawing analogies to biological evolution.
• Some commenters discuss the trade-offs between theoretical capacity and practical efficiency in LLM architectures.
• The role of human design in LLM architectures is questioned, with some suggesting that future architectures may be designed by machines rather than humans.