How AI Agents Learn Faster by Predicting Opponents in Parallel
This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents.
Original patent title: “Opponent modeling with asynchronous methods in deep RL”
This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents. Granted to Royal Bank of Canada in 2022 with 26 claims and 1 forward citation, and it is expected to expire in 2039.
Coverage
What does this patent actually cover?
This system trains a neural network for artificial intelligence agents using a method called parallelized asynchronous reinforcement learning, which means many 'worker' processors learn at the same time. Each worker interacts with a simulation, calculates how much its actions helped or hurt (local gradient computation), and then helps update the main AI's brain (global network parameters) (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1). Crucially, the system adds 'agent modeling' by including a special part in its learning calculation, called a 'supervised cross entropy loss' (Claim 1). This loss term helps the AI predict what other agents will do by comparing its predictions to what those agents actually did (observed one-hot encoded agent action) (Claim 2). For example, an AI learning to play a complex strategy game could use this to predict an opponent's next move, like where they might build a base or attack, making the learning agent smarter and faster to train.
The gap
What does this patent NOT cover?
- Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1, Claim 10).
- Does not cover AI training methods that do not use parallel processing with multiple worker processes (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1: 'parallelized asynchronous reinforcement learning').
- Does not cover learning systems where the loss function does not include at least a policy loss, a value loss, and a supervised cross entropy loss (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1).
- Does not cover agent modeling that doesn't involve comparing predicted actions against observed, one-hot encoded actions of other agents (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 2).
- Does not cover neural networks that do not learn opponent or teammate policies, potentially through a latent space (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 5, Claim 7).
These exclusions are unique to PatentBrief — derived from the actual claim language, not patent-office boilerplate.
Key facts
What made this novel
The clever part is integrating explicit opponent modeling directly into the asynchronous reinforcement learning process. Instead of just reacting to other agents, the AI actively tries to predict their behavior using a supervised loss term, which can significantly improve learning efficiency and the quality of the learned strategies.
The Patent Drawing
Schematic visualization of the patent's claim structure. Hand-drawn diagrams in progress for each landmark patent.
Where you've seen this
Real-world examples
AI for complex strategy games (e.g., StarCraft II, Dota 2)
Autonomous driving systems predicting other vehicle movements
Financial trading algorithms predicting market participant behavior
Robotics in collaborative or competitive multi-robot environments
AI agents for cybersecurity, predicting attacker actions
Why it matters
The bigger picture
This technology is important because it makes training complex artificial intelligence agents much faster and more effective, especially in environments where multiple AIs interact. By explicitly modeling opponents or teammates, the learning agent can develop more sophisticated strategies. The assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →, Royal Bank of Canada, suggests potential applications in areas like financial trading, where predicting the actions of other market participants is critical.
Filed
November 5, 2019
Granted
April 5, 2022
Market context
Who's building on this
Companies in this space
Major AI research labs like Google DeepMind and OpenAI, as well as companies developing AI for complex simulations and multi-agent systems, are actively working in this space. Financial institutions, including the Royal Bank of Canada, are also applying advanced AI techniques to areas like algorithmic trading and risk management, where predicting the actions of other entities is crucial.
Market impact
This patent contributes to the ongoing advancement of artificial intelligence, particularly in multi-agent learning environments. By improving the efficiency and effectiveness of training, it enables the development of more sophisticated AI agents capable of operating in complex, dynamic settings. This can lead to more robust AI solutions in industries ranging from finance and gaming to autonomous systems, potentially influencing competitive strategies and operational efficiencies.
Claim 1 — Plain English
What this patent covers
This system trains a neural network for artificial intelligence agents using a method called parallelized asynchronous reinforcement learning, which means many 'worker' processors learn at the same time. Each worker interacts with a simulation, calculates how much its actions helped or hurt (local gradient computation), and then helps update the main AI's brain (global network parameters) (Claim 1). Crucially, the system adds 'agent modeling' by including a special part in its learning calculation, called a 'supervised cross entropy loss' (Claim 1). This loss term helps the AI predict what other agents will do by comparing its predictions to what those agents actually did (observed one-hot encoded agent action) (Claim 2). For example, an AI learning to play a complex strategy game could use this to predict an opponent's next move, like where they might build a base or attack, making the learning agent smarter and faster to train.
The clever bit
The clever part is integrating explicit opponent modeling directly into the asynchronous reinforcement learning process. Instead of just reacting to other agents, the AI actively tries to predict their behavior using a supervised loss term, which can significantly improve learning efficiency and the quality of the learned strategies.
What it does not cover
- Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (Claim 1, Claim 10).
- Does not cover AI training methods that do not use parallel processing with multiple worker processes (Claim 1: 'parallelized asynchronous reinforcement learning').
- Does not cover learning systems where the loss function does not include at least a policy loss, a value loss, and a supervised cross entropy loss (Claim 1).
- Does not cover agent modeling that doesn't involve comparing predicted actions against observed, one-hot encoded actions of other agents (Claim 2).
- Does not cover neural networks that do not learn opponent or teammate policies, potentially through a latent space (Claim 5, Claim 7).
Patent timeline
Application submitted to the patent office
Application published, typically 18 months after filing
Patent officially issued
Patent enters public domain
PatentBrief Score
Impact Score
Moderate
Citation count
6/40
Early citations
Claim breadth
17/20
Very broad protection
Recency
20/20
Granted within 5 years
Assignee scale
0/20
Independent or smaller assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →
PatentBrief Impact Score — based on citation count, claim breadth, recency, and assignee scale. Not a legal assessment.
Heuristic Value Estimate
What this patent might be worth
$75K – $240K
Midpoint $150K · 13.2 yr remaining · industry ×1.6
Heuristic only — blends forward/backward citation counts, claim scope, time remaining, litigation history, and CPC-derived industry baseline. Real valuations need a professional appraisal.
Claim text not yet imported for this patent
The original legal language
Original claims
26 claims as filed with the patent office.
Concepts involved
Citations
Patent lineage
Cite this patent
LEAL, P. F. H., KARTAL, B., & TAYLOR, M. E. (2022). How AI Agents Learn Faster by Predicting Opponents in Parallel (U.S. Patent No. 11,295,174). U.S. Patent and Trademark Office. https://patentbrief.org/patent/us/11295174/opponent-modeling-with-asynchronous-methods-in-deep-rl
Auto-generated from the patent record. Double-check author order and the issue date against the official USPTO document before submitting.
Embed
Add this patent to your site
Drop this plain-English patent card into any blog post or article — free, no signup. It always links back to the full breakdown here.
<div data-patentlens-widget data-patent-number="US11295174"></div> <script src="https://patentbrief.org/embed.js" async></script>
Stay in the loop
Get a weekly digest of new patents.
One email per week. No spam. Unsubscribe anytime.
Keep exploring
Related patents you should know
US 4683195 · 1987
How to Make Billions of Copies of a DNA Segment
This patent describes the Polymerase Chain Reaction (PCR), a method to rapidly create many copies of a specific piece of DNA or RNA, enabling its detection and analysis.
Cetus Corp
US 8697359 · 2014
How to Edit Genes in Human Cells Using an Engineered CRISPR System
This patent describes an engineered CRISPR-Cas9 system for precisely cutting DNA in eukaryotic cells to change how genes work, opening the door for gene editing in complex organisms.
Massachusetts Institute of Technology
US 7657849 · 2010
How the iPhone's Slide-to-Unlock Gesture Works
Apple's 2010 patent describes unlocking a device by dragging a specific graphical image across the touchscreen along a predefined path, a gesture that became iconic with the original iPhone.
Apple Inc
US 4733665 · 1988
How Doctors Implant a Permanent Stent Using a Balloon
This patent describes the method for placing a permanent, expandable wire mesh tube inside a blood vessel or other body tube using a balloon-tipped catheter to widen it and keep it open.
Expandable Grafts Partnership
US 4965188 · 1990
How to Make Many Copies of a DNA Piece with Heat
This patent describes the Polymerase Chain Reaction (PCR) method, a technique to make millions of copies of a specific DNA segment using a heat-resistant enzyme and repeated temperature changes.
Cetus Corp
US 4235871 · 1980
How to Encapsulate Active Materials in Lipid Bubbles Efficiently
This patent describes a method for trapping biologically active substances inside tiny, multi-layered fat bubbles called liposomes, using a specific water-in-oil emulsion and gel-forming process to improve how much material gets captured.
Individual
Semantically similar
You might also find these interesting
US 11170293 · 2021 · Microsoft Technology Licensing LLC
How AI Systems Learn to Predict and Act Simultaneously
US 11429762 · 2022 · Amazon Technologies
How Computers Train AI Models Using Separate Virtual Simulations
US 12518214 · 2026 · Nant Holdings IP
Training AI on Private Data Without Seeing It
US 11836577 · 2023 · Amazon Technologies
Training Robot AI Models Faster Using Smart Simulations
More to explore
More in AI & Machine Learning
US 10452978 · 2019 · Google LLC
How AI Models Understand Language Using 'Attention'
US 6523026 · 2003 · Huntsman International LLC
How Computers Find Hidden Connections Between Different Fields of Knowledge
US 11615208 · 2023 · Capital One Services LLC
How Cloud Systems Automatically Create and Train AI Data Models
US 10402750 · 2019 · Facebook Inc
How Facebook Uses Deep Learning to Predict What You Might Like
New to patents?
Common Questions
Frequently Asked Questions
What does How AI Agents Learn Faster by Predicting Opponents in Parallel cover?
This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents.
Who owns patent US 11295174?
Royal Bank of Canada owns this patent, granted in 2022.
When does this patent expire?
This patent is expected to expire on November 5, 2039, when the invention enters the public domain.
What is patent US 11295174 cited by?
This patent has been cited by 1 later patents that build on its ideas.
What problem does this patent solve?
This technology is important because it makes training complex artificial intelligence agents much faster and more effective, especially in environments where multiple AIs interact. By explicitly modeling opponents or teammates, the learning agent can develop more sophisticated strategies. The assignee, Royal Bank of Canada, suggests potential applications in areas like financial trading, where predicting the actions of other market participants is critical.
What does this patent NOT cover?
Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (Claim 1, Claim 10).
Patent monitoring