How to Build a 3D Model That Separates Light from Objects

This patent describes a method to create a detailed 3D digital model of a real scene that can tell the difference between the light in the scene and the objects it illuminates, even transparent ones.

What it covers

The patent describes generating a "volumetric scene model" of a real-world location. This model is built from "digital images" captured by a camera from different viewpoints. A "scene reconstruction engine" processes these images to "disentangle light and matter fields." This means the resulting 3D model stores separate information about the physical properties of objects (the "matter field") and how light moves through the scene (the "light field"). The model includes "volumetric data elements" that represent material characteristics and "data elements" that represent light flow. A key feature is its ability to represent "partially light transmissive" materials, like glass or fog, by forming "corridors" between adjacent volumetric data elements. For example, a system could scan a room with a window and create a model that understands the glass is transparent and how light passes through it, separate from the solid walls.

What it doesn't cover

• —Does not cover scene reconstruction methods that only produce surface meshes without volumetric data representing the interior of objects or spaces.
• —Does not cover methods that do not explicitly separate (disentangle) the light field from the matter field within the generated scene model.
• —Does not cover systems that generate scene models primarily from non-image input data, such as pure LiDAR point clouds lacking associated light information.
• —Does not cover methods that cannot represent or account for partially light transmissive materials within the scene model, as specified by the "corridors" of transmissive media.
• —Does not cover real-time rendering of scenes without first generating and storing a volumetric scene model as described.

The clever bit

The truly novel aspect is the explicit disentanglement of the "light field" (how light moves) from the "matter field" (object properties) within a volumetric model, particularly its ability to handle partially light transmissive materials like glass or fog.

Why it matters

Understanding how light interacts with objects, especially transparent ones, is crucial for creating realistic digital representations of the world. This technology is foundational for advanced virtual reality (VR) and augmented reality (AR) experiences, where digital content needs to blend seamlessly with the real environment. It also has applications in film production for visual effects and in creating highly accurate digital twins for simulations and planning.

Real-world examples

• 1.Advanced photogrammetry software
• 2.Digital twin creation for industrial applications
• 3.High-fidelity virtual reality environments
• 4.Augmented reality applications that integrate virtual objects into real scenes
• 5.Visual effects pipelines for film and television