Running Stable Diffusion XL 1.0 in 298MB of RAM (github.com)

Article summary

The article discusses OnnxStream, a lightweight inference library that can run Stable Diffusion models on devices with limited RAM, such as the Raspberry Pi Zero 2. OnnxStream achieves this by decoupling the inference engine from the model weights and using techniques like attention slicing and dynamic quantization. The library can consume up to 55x less memory than other frameworks while being only slightly slower. This allows for running complex models like Stable Diffusion XL 1.0 on devices that would otherwise be unable to handle them.

Main themes

• OnnxStream library
• Stable Diffusion models
• Memory optimization
• Inference speed
• Model deployment
• Technical communication
• Memory bandwidth limitations
• Embedded devices

What commenters say

• The author's claim of OnnxStream being 0.5-2x slower than other frameworks is unclear and may be misleading.
• The library's ability to reduce memory consumption could be beneficial for running multiple models on a single GPU.
• The technique used by OnnxStream may not be effective for increasing throughput via larger batch sizes due to memory bandwidth limitations.
• The library's potential applications include running embedded models in digital devices like photo frames or wall paintings.
• The use of percentages to describe speed or time changes can be confusing and prone to misinterpretation.
• OnnxStream's approach could be useful for training models, even if it doesn't improve inference speed.
• The library's ability to run complex models on low-power devices could enable new use cases and applications.