AI PCs Aren't Good at AI: The CPU Beats the NPU (github.com)

Article summary

A benchmarking test on a Microsoft Surface tablet with a Qualcomm Arm-based SoC, marketed as an AI PC, showed that the NPU's performance was significantly lower than expected, achieving only 1.3% of its claimed 45 Teraops/s. The test compared the performance of the NPU and CPU on a simple matrix multiplication model, with the CPU outperforming the NPU. The results suggest that the NPU may have limited memory bandwidth or other performance bottlenecks. The benchmarking code and results are available on GitHub.

Main themes

• AI acceleration
• NPU performance
• Memory bandwidth
• Benchmarking
• AI marketing claims
• Hardware design

What commenters say

• The NPU's poor performance may be due to its memory interface or bandwidth limitations, rather than any inherent flaw in the hardware itself.
• The marketing claims of AI PCs are often exaggerated and do not reflect real-world performance, leading to disappointment and mistrust among consumers.
• The inclusion of NPUs in modern chips may be more of a marketing gimmick than a genuinely useful feature, given their limited performance and potential to increase power consumption.
• The design of modern chips is often constrained by thermal and power limitations, leading to trade-offs between performance, power consumption, and die size.
• Some commenters believe that the focus on AI acceleration and NPUs is misplaced, and that other features such as die size, power efficiency, and CPU performance are more important for real-world applications.
• Others argue that the NPU's performance is not the only factor to consider, and that its potential benefits in terms of energy efficiency and sustained performance may still make it a worthwhile feature.