Creating Your Own Image in Virtual and Augmented Reality

This patent describes how an artificial reality system can show a user their own image, called a self-representation, within a virtual world by using machine learning to identify and display parts of their live image.

What it covers

Based on the abstract, this patent describes a system that creates a digital version of a user, called a self-representation, for artificial reality environments. It works by taking an image of the user and then using a special computer program, a machine learning model, to find the 'self portion' of that image. This model has been taught to recognize parts of images that show a user from their own perspective. Once identified, this self-representation is placed into the artificial reality world, adjusted to match the user's viewpoint. The system also tracks the user's movements and updates their digital self-representation to move along with them, making it more realistic. For example, if a user is in a virtual meeting, this system could display a live, moving image of their face as their avatar.

What it doesn't cover

• —Does not cover creating a self-representation purely from a 3D model generated without an initial image of the user.
• —Does not cover self-representations that are not based on classifying a 'self portion' from an image using a machine learning model.
• —Does not cover systems where the self-representation is static and does not adjust to the user's real-time movements.
• —Does not cover displaying generic avatars or pre-made character models that do not originate from the user's live image.
• —Does not cover methods of self-representation that do not involve positioning the representation relative to the user's perspective in the artificial reality environment.

The clever bit

The clever part is using a machine learning model specifically trained to classify a 'self portion' from an image based on a 'self-perspective'. This allows the system to intelligently extract and display the relevant part of the user's image, rather than simply showing a raw video feed, and then dynamically adjust it within the artificial reality environment.

Why it matters

Accurate and dynamic self-representations are crucial for making artificial reality experiences feel real and immersive. This technology could enhance social interactions in virtual spaces, allowing users to see a more authentic version of themselves. It improves the sense of presence and connection in applications ranging from virtual meetings to collaborative design environments. The ability to dynamically update the representation with user movement makes interactions more natural.

Real-world examples

• 1.Meta Quest avatars with camera passthrough
• 2.Microsoft Mesh for collaborative virtual spaces
• 3.Apple Vision Pro persona generation
• 4.Zoom or Google Meet virtual backgrounds that integrate the user's image
• 5.Any future augmented or virtual reality system requiring realistic user avatars