# How AI Agents Learn Faster by Predicting Opponents in Parallel > This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents. - **Patent:** US 11295174 - **Original title:** Opponent modeling with asynchronous methods in deep RL - **Owner:** Royal Bank of Canada - **Granted:** 2022 - **Status:** Active - **Times cited:** 1 - **Field:** ai_ml, software, finance, telecommunications, gaming ## What it does This system trains a neural network for artificial intelligence agents using a method called parallelized asynchronous reinforcement learning, which means many 'worker' processors learn at the same time. Each worker interacts with a simulation, calculates how much its actions helped or hurt (local gradient computation), and then helps update the main AI's brain (global network parameters) (Claim 1). Crucially, the system adds 'agent modeling' by including a special part in its learning calculation, called a 'supervised cross entropy loss' (Claim 1). This loss term helps the AI predict what other agents will do by comparing its predictions to what those agents actually did (observed one-hot encoded agent action) (Claim 2). For example, an AI learning to play a complex strategy game could use this to predict an opponent's next move, like where they might build a base or attack, making the learning agent smarter and faster to train. ## What it does NOT cover - Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (Claim 1, Claim 10). - Does not cover AI training methods that do not use parallel processing with multiple worker processes (Claim 1: 'parallelized asynchronous reinforcement learning'). - Does not cover learning systems where the loss function does not include at least a policy loss, a value loss, and a supervised cross entropy loss (Claim 1). - Does not cover agent modeling that doesn't involve comparing predicted actions against observed, one-hot encoded actions of other agents (Claim 2). - Does not cover neural networks that do not learn opponent or teammate policies, potentially through a latent space (Claim 5, Claim 7). ## The clever bit The clever part is integrating explicit opponent modeling directly into the asynchronous reinforcement learning process. Instead of just reacting to other agents, the AI actively tries to predict their behavior using a supervised loss term, which can significantly improve learning efficiency and the quality of the learned strategies. ## Real-world examples 1. AI for complex strategy games (e.g., StarCraft II, Dota 2) 2. Autonomous driving systems predicting other vehicle movements 3. Financial trading algorithms predicting market participant behavior 4. Robotics in collaborative or competitive multi-robot environments 5. AI agents for cybersecurity, predicting attacker actions ## Why it matters This technology is important because it makes training complex artificial intelligence agents much faster and more effective, especially in environments where multiple AIs interact. By explicitly modeling opponents or teammates, the learning agent can develop more sophisticated strategies. The assignee, Royal Bank of Canada, suggests potential applications in areas like financial trading, where predicting the actions of other market participants is critical. ## Frequently asked questions ### What does How AI Agents Learn Faster by Predicting Opponents in Parallel cover? This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents. ### Who owns patent US 11295174? Royal Bank of Canada owns this patent, granted in 2022. ### When does this patent expire? This patent is expected to expire on November 5, 2039, when the invention enters the public domain. ### What is patent US 11295174 cited by? This patent has been cited by 1 later patents that build on its ideas. ### What problem does this patent solve? This technology is important because it makes training complex artificial intelligence agents much faster and more effective, especially in environments where multiple AIs interact. By explicitly modeling opponents or teammates, the learning agent can develop more sophisticated strategies. The assignee, Royal Bank of Canada, suggests potential applications in areas like financial trading, where predicting the actions of other market participants is critical. ### What does this patent NOT cover? Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (Claim 1, Claim 10). **Full plain-English explainer:** https://patentbrief.org/patent/us/11295174/opponent-modeling-with-asynchronous-methods-in-deep-rl **Original patent:** https://patents.google.com/patent/US11295174 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [How AI Systems Learn to Predict and Act Simultaneously](https://patentbrief.org/patent/us/11170293/alphago-policy-and-value-networks) — A method for training AI models that combines supervised learning for prediction with reinforcement learning for decision-making in a single, coordinated system. - [How Computers Train AI Models Using Separate Virtual Simulations](https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models) — This patent describes a system where one virtual computer runs simulations of a system, like a robot, and another virtual computer uses the simulation data to teach an AI model how to make better decisions. - [Training AI on Private Data Without Seeing It](https://patentbrief.org/patent/us/12518214/distributed-machine-learning-systems-including-generation-of-synthetic-data) — This patent describes a way to train artificial intelligence models using private data stored on many separate computers, by generating fake data that mimics the real data's patterns, so the private data itself never leaves its original location. - [Training Robot AI Models Faster Using Smart Simulations](https://patentbrief.org/patent/us/11836577/reinforcement-learning-model-training-through-simulation) — This patent describes a cloud service that helps train artificial intelligence models for robots by running simulations, even suggesting improvements to the AI's learning rules before starting. - [How AI Models Learn Complex Tasks by Breaking Them Down](https://patentbrief.org/patent/us/11120365/for-hierarchical-decomposition-deep-reinforcement-learning-for-an-artificial-int) — This patent describes a method for training artificial intelligence models to solve big problems by splitting them into smaller, manageable steps and learning each step separately before combining them.