What can LLMs never do? (strangeloopcanon.com)

Article summary

The article discusses the limitations of Large Language Models (LLMs) and their inability to perform certain tasks, such as playing Wordle or predicting cellular automata. Despite their impressive capabilities, LLMs struggle with tasks that require iterative reasoning, memory, and dynamic attention. The author suggests that these limitations are due to the fundamental architecture of LLMs and may not be overcome by simply increasing the size of the models or the amount of training data. The article explores the implications of these limitations for the development of Artificial General Intelligence (AGI).

Main themes

• LLM limitations
• Iterative reasoning
• Dynamic attention
• AGI development
• Language modeling
• Cognitive architectures

What commenters say

• LLMs will never be able to perform certain tasks, regardless of their size or training data, due to their fundamental architecture.
• The limitations of LLMs can be overcome by increasing their size and training data, and they will eventually become capable of performing any task.
• LLMs are not truly intelligent, but rather sophisticated predictive models that are prone to hallucinations and errors.
• The development of AGI will require a fundamentally different approach than the current LLM architecture, which is limited by its lack of dynamic attention and iterative reasoning.
• The ability of LLMs to perform certain tasks is not a matter of intelligence, but rather a result of their training data and algorithms.
• The limitations of LLMs are not unique to AI models, but are also present in human cognition, and therefore should not be seen as a major obstacle to AGI development.
• The concept of cognition is not well understood, and it is unclear whether LLMs or any other AI models are truly cognitive or not.
• The hallucination problem in LLMs is a significant issue that needs to be addressed, but it can be mitigated by building larger and better models.