Running local models is good now (vickiboykis.com)

Article summary

The author discusses their experience with running local models, which have improved significantly in recent months, allowing for fast and personalized development tasks. They mention using various models, including Gemma 4 and Qwen 3, on a 2022 M2 Mac with 64 GB RAM and 1TB storage. The author notes that while local models still have limitations, such as slow inference and small context windows, they have become more viable for development tasks. The author also shares their setup and configuration for running local models using Docker and LM Studio.

Main themes

• Local models
• AI development
• Hardware requirements
• Cloud vs local
• Cost and affordability
• Model performance

What commenters say

• The cost of hardware for running local models is prohibitively expensive for many individuals, making cloud services a more accessible option.
• Some professionals can justify the cost of high-end hardware for local models due to the potential return on investment and increased productivity.
• Local models may not offer significant advantages over cloud services for coding tasks, and the cost of hardware may not be justified for many users.
• The ability to run local models could disrupt the business model of companies that rely on renting AI services, as users may opt to buy and run their own models instead.
• There is a trade-off between the convenience and scalability of cloud services and the control and security of running local models.
• The development of local models and related tools is rapidly improving, making them more viable for a wider range of users and applications.
• Some companies may prefer to run their own on-premises AI clusters due to trust and control concerns, rather than relying on cloud services.
• The trend towards outsourcing and cloud computing may continue with AI, with many businesses opting to pay a premium for convenience and scalability rather than managing their own infrastructure.