Llama.cpp 30B runs with only 6GB of RAM now (github.com)

Article summary

The Llama.cpp project has achieved a significant optimization, allowing the 30B model to run with only 6GB of RAM. This was accomplished by changing the file format to enable direct mapping of weights into memory, reducing loading times and increasing the potential for concurrent inference processes. The new format also supports single-file and multi-file models, and ensures proper tensor alignment for potential performance gains. The project's developers are working to ensure a smooth transition for users of the old file format.

Main themes

• AI optimization
• open-source collaboration
• distributed computing
• proprietary models
• barriers to entry
• GPT-4 applications
• FOSS community resources

What commenters say

• The open-source community is achieving rapid optimizations of large language models, potentially undermining the competitive advantage of proprietary models.
• Hiring optimization experts could have prevented the significant depreciation of proprietary models due to open-source advancements.
• The high minimum requirements for AI training create a barrier to entry for hobbyists and small organizations, but collaborative efforts may be able to overcome this.
• Leveraging distributed computing, such as through a SETI@home-style project, could be a viable way to train large open-source models.
• The use of GPT-4 and other AI tools may be able to accelerate the development of new technologies, including those related to AI training.
• The FOSS community lacks a mechanism to pool resources for large-scale computing projects, but crowdfunding or partnerships with cloud providers could be potential solutions.