# How Computers Train AI Models Using Separate Virtual Simulations

> This patent describes a system where one virtual computer runs simulations of a system, like a robot, and another virtual computer uses the simulation data to teach an AI model how to make better decisions.

- **Patent:** US 11429762
- **Original title:** Simulation orchestration for training reinforcement learning models
- **Owner:** Amazon Technologies
- **Granted:** 2022
- **Status:** Active
- **Times cited:** 13
- **Field:** software, ai_ml, telecommunications, robotics, automotive

## What it does

The patent outlines a method for training a reinforcement learning model by orchestrating two virtual compute nodes. A 'simulation workflow manager' (from the abstract) configures a first virtual computer with a training application and a second virtual computer with a simulation application (claim 1). The second computer runs a simulation of a system, such as a 'robotic device' (claim 1), which performs actions chosen by the model. Data from this simulation, including the action taken, the resulting new state, and a reward value (claim 3), is then sent to the first virtual computer. The training application on the first computer uses this data to improve the model. This process can repeat, with the updated model being sent back to the simulation for further refinement (claim 4).

## What it does NOT cover

- Training AI models without using a separate, dedicated simulation environment.
- Simulations that do not involve a 'robotic device' performing actions in the simulation environment (claim 1).
- Training AI models where the model does not randomize the selection of actions within the simulation (claim 1).
- Training AI models where the simulation and the model training occur on the same single compute node.
- Simulations performed for purposes other than training a reinforcement learning model for system optimization.

## The clever bit

The novelty lies in the orchestrated separation of the computationally intensive simulation from the model training onto distinct, configurable virtual compute nodes. This allows for specialized scaling and optimization of each part of the reinforcement learning loop, with iterative data exchange between them.

## Real-world examples

1. AWS RoboMaker
2. Autonomous vehicle simulation platforms
3. Industrial robot training systems
4. Drone navigation AI training
5. Cloud-based machine learning platforms

## Why it matters

This approach allows for efficient and scalable training of complex AI models, especially for robotics or autonomous systems, without needing expensive physical hardware for initial training. It enables rapid iteration and testing of AI behaviors in a safe, virtual environment. The separation of simulation and training tasks allows each component to be optimized and scaled independently, speeding up development.

## Frequently asked questions

### What does How Computers Train AI Models Using Separate Virtual Simulations cover?

This patent describes a system where one virtual computer runs simulations of a system, like a robot, and another virtual computer uses the simulation data to teach an AI model how to make better decisions.

### Who owns patent US 11429762?

Amazon Technologies owns this patent, granted in 2022.

### When does this patent expire?

This patent is expected to expire on November 27, 2038, when the invention enters the public domain.

### What is patent US 11429762 cited by?

This patent has been cited by 13 later patents that build on its ideas.

### What problem does this patent solve?

This approach allows for efficient and scalable training of complex AI models, especially for robotics or autonomous systems, without needing expensive physical hardware for initial training. It enables rapid iteration and testing of AI behaviors in a safe, virtual environment. The separation of simulation and training tasks allows each component to be optimized and scaled independently, speeding up development.

### What does this patent NOT cover?

Training AI models without using a separate, dedicated simulation environment.

**Full plain-English explainer:** https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models

**Original patent:** https://patents.google.com/patent/US11429762

---

_Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._