SMERF: Streamable Memory Efficient Radiance Fields (smerf-3d.github.io)

Article summary

The article introduces SMERF, a view synthesis approach that achieves state-of-the-art accuracy among real-time methods on large scenes. SMERF is built upon a hierarchical model partitioning scheme and a distillation training strategy, enabling full six degrees of freedom navigation within a web browser and rendering in real-time on commodity devices. The approach exceeds the current state-of-the-art in real-time novel view synthesis and achieves real-time performance across a wide variety of devices. The SMERF models are distilled from Zip-NeRF checkpoints trained on various scenes.

Main themes

• Real-time view synthesis
• Neural fields and radiance fields
• 3D reconstruction and rendering
• SMERF and Zip-NeRF
• Mirror worlds effect
• Code release and open-sourcing
• Performance and optimization
• Limitations and challenges

What commenters say

• The SMERF demo is surprisingly smooth and high-fidelity on older smartphones.
• The codebase is heavily wired into other not-yet-open-sourced libraries, making it difficult to release the code immediately.
• The mirror worlds effect in NeRFs can create a spooky and immersive experience, but is a common issue with 3D reconstruction.
• The SMERF approach has limitations, such as spatial resolution and teacher model quality, which can affect reconstruction quality.
• The large payload sizes of SMERF can lead to long loading times, but breaking down the data into smaller chunks could help improve the experience.
• The SMERF viewer biases towards assets closer to the user's camera, but training with a larger number of smaller submodels can become too onerous and affect quality.
• The mirror worlds effect is not unique to SMERF and can be seen in other NeRFs and even game engines.
• Compressing the payload sizes by 10x could significantly improve the SMERF experience.