It’s NeRFs or nothing, the rapid development of Neural Radiance Fields

What is a Neural Radiance Field (NeRF)?

A NeRF is a neural network for generating views of complex 3D scenes based on a partial set of 2D images. NeRF takes anywhere from a dozen to hundreds of input images that show a scene and interpolates between them to render a complete 3D-modelled scene. This modelled scene contains features such as texturing, shading, shadows, lighting, and viewpoints that enable even more advanced uses.

The technology is one of the many up-and-coming innovations making use of deep learning and since their original proposal in 2020 has seen an explosion of papers, research, and advancements. Neural fields even recently saw a feature in Time magazine’s ‘best inventions of 2022’ list.

By using neural fields in place of traditional render methods such as voxel grids or polygon meshes, they can be highly efficient and compact 3D representations of objects that are also differentiable and continuous. An additional advantage of this is their ability to have arbitrary dimensions and resolutions as well as being domain agnostic, meaning they do not depend on the original input for each task massively cutting down on resource usage.

Once trained, the neural fields can be used to produce a NeRF for view synthesis. This is the method of generating a 3D object or scene from a set of pictures from different viewpoints and angles. Effectively, it is an advanced method of 3D reconstruction.

The applications of NeRF

The major application of NeRF primarily lies in its ability to render 3D models or scenes in a matter of moments, something that was practically inconceivable only a few years ago. This ability to rapidly render 3D models and scenes means that in the not-so-distant future, NeRF could see active use in the film, simulation, or gaming industries for example. What could have potentially taken teams of people weeks to produce can be replicated in just a day’s work and simply touched up to perfect it.

In film, scenes or sets that need to be 3D modelled for CG use later down the pipeline could be scanned and rendered in a far more time and cost-efficient way. Simulations could use the tech in a similar way to scan hyper-realistic environments for use in simulations. As for gaming, it could be used to cut down on costs and time relating to the environment and world-building, potentially revolutionising the industry.

Expanding on NeRF

Since the explosion of the popularity of NeRF in 2022, countless branches expanding on the base NeRF formula have been gaining traction all with the aim to improve different areas of NeRF performance. RegNeRF, pixelNeRF, Mega-NeRF, LOLNeRF, NSVF, Mip-NeRF, KiloNeRF, and Plenoxels are all great examples of these branches. ReGNeRF, for example, takes the base method and adds regularising for view synthesis from a sparse number of inputs. By doing so, it helps to address the problem of poor performance and render quality if the number of 2D image input views is low. Mega-NeRF on the other hand is an alternative deep learning framework that aims to expand the scale of NeRFs allowing them to scan and build large-scale interactive environments such as buildings or even entire city blocks.

The future of NeRF

So what lies in the future for NeRF? To put it plainly, more attention and more developments. NeRF is a brand-new innovation on the very edge of making some serious breakthroughs. Within the later stages of 2022 and early 2023 alone, dramatic progress has been made to make the technology more accessible and advanced. Nvidia’s NGP Instant NeRF allows you at home with a smartphone and a light render-capable PC to create your very own NeRF to tinker with to your heart’s content. Meanwhile, even areas such as the Metaverse are taking note, with researchers at Meta experimenting with AI and NeRFs to create highly realistic rendered avatars and interfaces, calling them ‘Codec Avatars’.

NeRFs are one of the most exciting emerging technologies of the last few years. The ability to render 3D models in a simplistic manner on a small budget, and in a relatively small timeframe is something that should not be shrugged off lightly. It will not be long until we see NeRFs in the mainstream.