Nvidia’s latest graphics research takes images from 2D to 3D almost instantly | ZDNet

The metaverse is taking shape right now, and Nvidia has gone all in, introducing a robust set of tools to build it. But even for a graphics pioneer like Nvidia, rendering 3D worlds is a complicated technical challenge. 

At its spring Graphics Technology Conference (GTC) this week, Nvidia demonstrated a new approach to inverse rendering — the process of reconstructing 3D scenes from a handful of 2D images. Inverse rendering uses AI to approximate how light behaves in the real world. With the approach developed by the Nvidia Research team, the whole process happens almost instantly. 

The method, Nvidia explained, could be applied to a wide range of use cases. It could be used to create avatars or scenes for virtual worlds, to capture video conference participants and their environments in 3D, or to reconstruct scenes for 3D digital maps. 

Using traditional methods to create a 3D scene can take hours or longer, depending on the complexity and resolution of the visualization. A popular new technology called neural radiance fields (NeRF) uses AI to speed up the process. NeRFs use neural networks to represent and render realistic 3D scenes based on an input collection of 2D images. A NeRF effectively trains a small neural network to fill in the blanks by predicting the color of light radiating in any direction from any point in 3D space. 

Early NeRF models have produced quality renderings in a few minutes but have taken hours to train. Nvidia Research has developed what it calls Instant NeRF — combining fast neural network training with rapid rendering. It’s the fastest NeRF technique to date, Nvidia says, achieving more than 1,000x speedups in some cases. 

To develop Instant NeRF, Nvidia came up with a new input encoding method called multi-resolution hash grid encoding. It’s optimized to run efficiently on Nvidia GPUs.

“Instant NeRF could be as important to 3D as digital cameras, and JPEG compression have been to 2D photography — vastly increasing the speed, ease and reach of 3D capture and sharing,” David Luebke, VP for graphics research at Nvidia, said in a statement.