Jamesob's guide to running SOTA LLMs locally (github.com)

Article summary

The article discusses a guide to running state-of-the-art large language models (LLMs) locally, with a focus on hardware setup and configuration. The author shares their experience with building a system using 4x RTX 6000 Pro cards, which allows for running models like Qwen3.6-27B. The guide also covers topics such as PCIe switches, power limiting, and running models using Docker containers. The goal is to provide a cost-effective solution for running LLMs locally, with options ranging from $2k to $40k.

Main themes

• Local LLM setup
• Hardware configuration
• GPU performance
• Cost-effectiveness
• Model performance
• Memory bandwidth

What commenters say

• Running LLMs locally can be a cost-effective solution, with options available for different budgets, but may require significant upfront investment.
• The choice between using a MacBook or a custom-built GPU setup depends on factors such as memory bandwidth, token generation speed, and usability.
• Some commenters argue that MacBooks can become unusable when running local models due to poor implementation, while others find them sufficient for certain tasks.
• The use of PCIe switches and power limiting can significantly impact the performance of LLMs, and proper configuration is crucial for optimal results.
• There is a trade-off between model size and performance, with larger models requiring more resources but potentially offering better results, and some commenters argue that smaller models can be more effective in certain situations.
• The cost of GPUs is expected to remain high for the next 18-24 months, making it challenging for individuals to build or upgrade their systems, and some commenters are skeptical about the possibility of significant price drops in the near future.