Why I'm still bearish on LLMs after Navier-Stokes (dank.systems)

Article summary

The article discusses the limitations of current large language models (LLMs) and why they are not yet ready for fully autonomous use in most domains. The author argues that LLMs require laborious oversight and guardrails, and that their ability to generalize is limited to tasks within a small neighborhood of their training data. The article also highlights the problem of reward hacking and the need for rigorous specification by domain experts. The author concludes that LLMs will likely be used in a limited capacity, with human review and oversight, rather than as a fully autonomous replacement for knowledge workers.

Main themes

• LLM limitations
• automation and augmentation
• rigorous specification and validation
• human review and oversight
• domain expertise
• learning and generalization
• structural and cost limitations
• future of work and automation

What commenters say

• LLMs are useful but overpriced and not yet ready for fully autonomous use.
• The limitations of LLMs are not just technical, but also structural and related to the cost of rigorous specification and validation.
• Some commenters argue that LLMs will still have a significant impact on automation, even if they are not perfect.
• Others believe that LLMs will fundamentally change how we approach domain expertise, but will not replace human experts entirely.
• There is disagreement about whether LLMs can learn and generalize in the same way as humans, with some arguing that they are limited by their training data and others pointing to examples of successful learning and adaptation.
• The use of LLMs in specific domains, such as customer support, is seen as potentially beneficial but also limited by the need for human review and oversight.
• Some commenters are skeptical about the potential for LLMs to replace human workers, citing the importance of human judgment and expertise in many domains.
• Others believe that LLMs will enable new forms of automation and augmentation, but will require significant advances in areas such as context management and scaffolding.