news.volyx.in

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac (github.com)

919 points by gitpusher42 · 29 days ago · 345 comments on HN

Article summary

The article presents TurboFieldfare, an open-source engine that runs the Gemma 4 26B-A4B model in approximately 2 GB of RAM on any M-series Mac. This is achieved by streaming the required model weights from SSD, rather than loading the entire 14.3 GB model into memory. The engine is written in Swift and Metal and provides a native Mac app, command-line interface, and experimental loopback OpenAI-compatible server. The project aims to enable text-only inference on Apple Silicon Macs with limited RAM.

Main themes

  • TurboFieldfare engine
  • Gemma 4 26B-A4B model
  • Memory efficiency
  • SSD performance
  • Caching and streaming
  • Model comparison
  • Portability and compatibility
  • Performance optimization

What commenters say

  • The engine's performance is significantly affected by the system's caching and SSD performance, with faster SSDs and more available RAM resulting in better performance.
  • The use of expert caching and streaming from SSD allows the engine to run the model in a relatively small amount of RAM, making it suitable for devices with limited memory.
  • Some commenters question the comparison to other models and engines, such as DwarfStar4 and llama.cpp, and suggest that a more thorough evaluation is needed to determine the strengths and weaknesses of TurboFieldfare.
  • The engine's ability to run on devices with limited RAM is seen as a major advantage, but some commenters note that this may come at the cost of slower performance compared to other approaches.
  • There is interest in porting the engine to other platforms, such as Windows and iOS, but this would require significant modifications to the underlying architecture.
  • Some commenters discuss the potential for the engine to cause wear and tear on the SSD, but others point out that reads do not wear out flash memory to a significant extent.
  • The engine's performance is expected to throttle on fanless devices, such as the MacBook Air, due to the intense computational workload.
  • The use of Metal and Apple's shared memory architecture is seen as a key factor in the engine's performance, but this also limits its portability to other platforms.