iPhone 17 Pro Demonstrated Running a 400B LLM (twitter.com)

Article summary

The iPhone 17 Pro has been demonstrated running a 400B large language model (LLM), a feat that was previously considered impossible. This achievement is attributed to both hardware and software advancements. The iPhone's hardware, including its Neural Engine and tensor processing on the GPU cores, has improved significantly, while software engineers have also made notable progress in optimizing LLMs for consumer hardware. The demonstration shows the potential for running large AI models on mobile devices.

Main themes

• LLM on mobile devices
• Hardware advancements
• Software optimization
• AI on consumer hardware
• Neural Engine and GPU cores

What commenters say

• The ability to run a 400B LLM on an iPhone is a significant achievement that was previously considered impossible due to hardware limitations.
• The feat is more a result of software triumph, where engineers have crafted a large model to run on consumer hardware, rather than a hardware breakthrough.
• The improvement in running LLMs on mobile devices is not just about raw power, but also about the ability to repurpose cheaper, lesser-scale hardware for inference in bulk.
• The quality of the output is degraded severely to achieve the speed, and the demonstration is more of a proof-of-concept rather than a practical application.
• The use of SSD streaming to GPU, as described in Apple's 'LLM in a flash' paper, is a key factor in enabling the iPhone to run large LLMs.
• The field of AI will continue to see advancements in architectural efficiency and information density, making it possible to run high-quality, real-time inference on mobile devices.
• The main bottleneck in running LLMs on mobile devices is storage bandwidth, and improving SSD speed can significantly enhance performance.
• The demonstration of running a 400B LLM on an iPhone has implications for the future of TinyML and EdgeML capabilities, with potential for commodification of SoCs comparable to the A19 Pro.