How to Combine Wide-Angle LiDAR Depth with Standard Camera Images

This patent describes a method for combining a wide-angle depth map from a LiDAR sensor, which has unevenly spaced pixels, with a standard camera image by first correcting the depth map's pixel distribution and then merging the two for a complete view.

What it covers

This patent outlines a method to create a detailed composite image by merging data from two different sensors. First, it captures a "first image" of a scene using a sensor like LiDAR, which measures depth and has pixels distributed unevenly, often non-linearly in the vertical direction (Claim 1, 2). This unevenness means some areas have more detail than others. It also captures a "second image" from a different sensor, such as a regular digital camera (Claim 1, 9), which typically has a more uniform pixel distribution. The key step is "correcting the first image" (Claim 1) based on its non-linear distribution function to produce a "third image," potentially by adding more pixels through interpolation (Claim 6, 7). Finally, it combines this corrected depth image (the third image) with the camera image (the second image) to create a "composite image" (Claim 1), such as an RGBD image (Claim 15), providing both color and accurate depth across a wide field of view (Claim 12). For example, an autonomous vehicle could use this to get a wide, detailed 3D view of its surroundings by merging a wide-angle LiDAR's depth data with a standard camera's color feed.

What it doesn't cover

• —Does not cover combining images where both sensors already have a linear or uniform pixel distribution.
• —Does not cover systems that combine images without first correcting the non-linear pixel distribution of one sensor.
• —Does not cover systems that only use a single sensor to generate a composite image.
• —Does not cover combining images where neither the first nor the second image is a depth map.
• —Does not cover combining images from two sensors that do not have at least one common field of view.

The clever bit

The novelty lies in specifically addressing the challenge of integrating wide-angle sensors, like LiDAR, that inherently produce images with non-uniform pixel distribution – meaning some areas have more detail than others – by correcting this distortion *before* merging it with a standard, uniformly distributed camera image. This ensures accurate alignment and a consistent data representation across the entire wide field of view.

Why it matters

This technology is important for applications like autonomous driving and robotics, where understanding the environment in 3D is critical. By effectively combining wide-angle depth information from LiDAR with high-resolution color data from cameras, systems can gain a more complete and accurate perception of their surroundings. This allows for better object detection, tracking, and navigation, especially in complex and dynamic environments requiring wide fields of view.

Real-world examples

• 1.Autonomous vehicle perception systems
• 2.Robotics navigation and obstacle avoidance
• 3.Advanced Driver-Assistance Systems (ADAS)
• 4.3D mapping and surveying
• 5.Augmented reality depth sensing