news.volyx.in

Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models (cerebras.net)

567 points by asb · 1252 days ago · 232 comments on HN

Article summary

Cerebras has released a family of open, compute-efficient, large language models called Cerebras-GPT, trained using the Chinchilla formula. The models range from 111 million to 13 billion parameters and are available on Hugging Face and GitHub under the Apache 2.0 license. Cerebras-GPT is designed to be used by and reproducible by anyone, with detailed information on training methods and performance results provided in a paper. The models were trained on CS-2 systems, part of the Andromeda AI supercomputer, using a simple, data-parallel weight streaming architecture.

Main themes

  • Large Language Models
  • Compute Efficiency
  • Open Source AI
  • Model Training
  • Inference Costs
  • AI Hardware
  • Naming Conventions
  • Technical Issues
  • Industry Trends
  • Company Strategies

What commenters say

  • Some commenters question the usefulness of smaller models, such as the 111 million parameter model, and suggest that scaling up the training data might be more beneficial.
  • Others argue that good zero-shot performance is a good starting point for fine-tuning, and that optimizing for zero-shot performance can be beneficial for multiple inference use cases.
  • There is disagreement on whether the Chinchilla formula is relevant for models with intended high inference use cases, with some arguing that it ignores the inference part of the equation.
  • The cost of training versus inference is discussed, with some arguing that the cost of inference will dwarf the cost of training, while others point out that this may not be the case for companies that need to fine-tune or train their own models.
  • The naming of models, such as Chinchilla, Alpaca, and LLaMA, is seen as part of a trend of using cute or exotic names for AI models.
  • Some commenters note that the images on the Cerebras website are blurry and of poor quality, which may be due to dynamic scaling or other technical issues.
  • The potential for other companies, such as Nvidia, to work on LLMs is discussed, with some noting that Nvidia's chips may not be optimized for LLMs due to RAM limitations.