Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU (github.com)

Article summary

The article presents a high-efficiency LLM inference engine in C++/CUDA that can run Llama 70B on a single RTX 3090, achieving 83x speedup over the baseline. The engine uses a 3-tier adaptive caching system and NVMe direct I/O to bypass the CPU. The project's goal is to enable fast and efficient inference of large language models on consumer hardware. The engine is open-source and available on GitHub.

Main themes

• LLM inference optimization
• GPU acceleration
• NVMe direct I/O
• C++/CUDA implementation
• Model quantization
• Memory bandwidth

What commenters say

• The current implementation is too slow for interactive use cases and may not be practical for many applications.
• The performance of the engine is bottlenecked by the PCIe bandwidth and memory bandwidth, rather than the GPU compute power.
• Using a smaller model with lower latency may be a better tradeoff for many use cases, rather than trying to run a large model on a single GPU.
• The idea of using NVMe direct I/O to bypass the CPU is promising, but may require significant modifications to the system and kernel.
• Some commenters suggest that running the model on a CPU with a large amount of RAM may be faster than using a GPU with limited VRAM.
• There is ongoing research in optimizing MoE models for inference, including using multi-tier caching and predicting layer swaps to maximize bandwidth.
• The engine's performance may be improved by using a faster GPU or a system with more advanced PCIe and memory capabilities.
• The tradeoff between model size, latency, and accuracy is a key consideration in designing and optimizing LLM inference engines.