A 10 year old Xeon is all you need (point.free)

Article summary

The article discusses running a 26B MoE model, Gemma 4, on an old recycled server with a single Intel Xeon E5-2620 v4 and 128GB of DDR3 RAM, without a GPU. The author achieved this by using the ik_llama-cpp fork and applying various optimizations, including speculative decoding, CPU and MoE routing, memory pinning, and graph layout adjustments. The model was able to run at reading speed, although the exact performance metrics were not provided. The author highlights the importance of understanding the optimization levers and flags used in the process.

Main themes

• Optimizing AI models on old hardware
• Memory-bound workloads
• CPU and MoE routing
• Graph layout and parallelism
• Energy efficiency and cost

What commenters say

• The article's claim of running the model at reading speed is questionable without proper benchmarking.
• Newer, more efficient hardware is often preferable to older systems due to better performance per watt.
• Recycling old hardware can be a viable and environmentally friendly option for certain use cases, despite potential energy efficiency drawbacks.
• The choice between using old or new hardware depends on various factors, including energy costs, performance requirements, and eco-friendliness.
• Some commenters disagree on the energy efficiency of older Xeon systems, with some reporting relatively low power consumption and others citing high energy usage.
• The article's website layout and readability were criticized, with suggestions for improvement.
• There is debate about the effectiveness of Simultaneous Multithreading (SMT) in memory-bound workloads, with some arguing it can help and others claiming it adds scheduling cost without adding throughput.