Running local models on an M4 with 24GB memory (jola.dev)

Article summary

The author has been experimenting with running local models on a 24GB M4 Macbook Pro and has found a setup that works reasonably well, using the Qwen 3.5-9B model with LM Studio. This setup allows for basic tasks, research, and planning without an internet connection, although it is not as powerful as state-of-the-art models. The author notes that local models have tradeoffs, but also benefits such as no internet connection required, limited cost, and environmental benefits. The author shares their experience with using the model for coding tasks, including examples of successes and failures.

Main themes

• Local AI models
• M4 Macbook Pro
• Qwen model
• LM Studio
• Coding tasks
• Environmental benefits

What commenters say

• Running local models on a 24GB M4 Macbook Pro can be a viable option for basic tasks, but may not be sufficient for more complex projects.
• The choice of model and quantization level can significantly impact the performance of local models, with some users reporting success with larger models and 6-bit quantization.
• Local models may not be able to match the performance of cloud-based models, but offer benefits such as control, privacy, and transparent cost models.
• The cost of a high-end laptop with sufficient RAM to run local models can be prohibitively expensive, and may not be justified by the potential savings on cloud subscriptions.
• Some users argue that local models are not yet ready for prime time, and that cloud-based models are still the better option for most use cases.
• Others argue that local models can be a useful tool for developers, especially those who value control and privacy, and that the benefits outweigh the costs.
• The performance of local models can be impacted by the specific hardware and software configuration, and some users report better results with certain models and quantization levels.
• The decision to use local models versus cloud-based models depends on individual needs and priorities, including factors such as cost, performance, and environmental concerns.