news.volyx.in

Flash-MoE: Running a 397B Parameter Model on a Laptop (github.com)

398 points by mft_ · 162 days ago · 119 comments on HN

Article summary

The article presents a project called Flash-MoE, which runs a 397 billion parameter Mixture-of-Experts model on a MacBook Pro with 48GB RAM, achieving 4.4+ tokens per second. The model uses a custom Metal compute pipeline and streams the 209GB model from an SSD. The project demonstrates the feasibility of running large models on relatively modest hardware. The model's performance is made possible by various optimizations, including SSD expert streaming and FMA-optimized dequant kernels.

Main themes

  • Mixture-of-Experts models
  • Model optimization
  • SSD storage
  • GPU acceleration
  • Metal compute pipeline
  • Large model inference

What commenters say

  • The project's results are impressive, but the performance may not be sufficient for heavy usage due to the SSD's limited bandwidth and potential wear and tear.
  • The use of SSD storage for model weights is a viable approach, but it may not be suitable for all use cases due to the potential for wear and tear.
  • The project's optimization techniques, such as FMA-optimized dequant kernels, are effective in improving model performance, but may not be applicable to all models or hardware configurations.
  • The performance of the model is bottlenecked by the SSD bandwidth, and using multiple SSDs or alternative storage solutions could potentially improve performance.
  • The project demonstrates the potential for running large models on modest hardware, but the performance may vary depending on the specific model and use case.
  • The use of system memory and CPU compute for some model layers is a viable approach, but may not be as effective as using GPU acceleration for certain models and use cases.
  • The project's approach could be applied to other models and hardware configurations, including mobile GPUs and Linux systems, with potential modifications and optimizations.
  • The trade-off between model performance and SSD wear and tear is a significant consideration, and alternative storage solutions or optimization techniques may be necessary for long-term usage.