Making Real-World Sounds Fit into Mixed Reality Experiences

This patent describes how a wearable device can take sounds from the real world, change them to match the acoustics of a virtual environment, and then play them back to a user, making mixed reality feel more realistic.

What it covers

This patent focuses on processing audio for mixed reality. It describes a system where a wearable device receives an audio signal from a real-world sound source. The system then figures out the acoustic properties of the mixed reality environment, such as how sound would echo or be muffled in that virtual space. Based on these properties, it calculates a signal modification parameter. This parameter is then applied to the original real-world audio signal, creating a new, modified audio signal. Finally, this modified audio signal is presented to the user, making real-world sounds seem like they are naturally part of the mixed reality scene. For example, if you're wearing a mixed reality headset and talking to someone in a real room, but the virtual environment shows you in a large, empty hall, this system would add echo to the person's voice to match the virtual hall.

What it doesn't cover

• —It does not cover audio processing for purely virtual reality environments where all sounds originate from the virtual world.
• —It does not cover augmented reality systems that simply overlay virtual objects without modifying real-world audio based on virtual acoustic properties.
• —It does not cover systems that modify real-world audio without first determining acoustic properties of a mixed reality environment.
• —It does not cover audio systems that do not use a wearable device to receive the input audio signal.
• —It does not cover modifying virtual sounds based on real-world acoustic properties; it specifically focuses on real-world sounds modified by mixed reality properties.

The clever bit

The novelty lies in dynamically altering real-world audio based on the *acoustic characteristics of a virtual environment*. Instead of just playing real sounds, it makes them sound as if they are actually interacting with the virtual space, which significantly enhances the feeling of presence and immersion in mixed reality.

Why it matters

Realistic audio is crucial for believable mixed reality experiences. If a virtual object sounds out of place in the real world, or a real sound doesn't match the virtual environment, it breaks the user's immersion. This technology helps blend the auditory experience between the physical and digital worlds, making interactions feel more natural. It's a foundational piece for creating compelling mixed reality applications that seamlessly integrate virtual elements with our physical surroundings.

Real-world examples

• 1.Meta Quest Pro
• 2.Apple Vision Pro
• 3.Microsoft HoloLens
• 4.Magic Leap headsets
• 5.Augmented reality glasses with spatial audio features