How AI Agents Learn Faster by Predicting Opponents in Parallel

This patent describes a computer system that trains artificial intelligence agents more efficiently by using many processors to simultaneously learn from a simulated environment, while also explicitly predicting the actions of other agents.

What it covers

This system trains a neural network for artificial intelligence agents using a method called parallelized asynchronous reinforcement learning, which means many 'worker' processors learn at the same time. Each worker interacts with a simulation, calculates how much its actions helped or hurt (local gradient computation), and then helps update the main AI's brain (global network parameters) (Claim 1). Crucially, the system adds 'agent modeling' by including a special part in its learning calculation, called a 'supervised cross entropy loss' (Claim 1). This loss term helps the AI predict what other agents will do by comparing its predictions to what those agents actually did (observed one-hot encoded agent action) (Claim 2). For example, an AI learning to play a complex strategy game could use this to predict an opponent's next move, like where they might build a base or attack, making the learning agent smarter and faster to train.

What it doesn't cover

• —Does not cover reinforcement learning systems that train a single AI agent without modeling other agents or opponents (Claim 1, Claim 10).
• —Does not cover AI training methods that do not use parallel processing with multiple worker processes (Claim 1: 'parallelized asynchronous reinforcement learning').
• —Does not cover learning systems where the loss function does not include at least a policy loss, a value loss, and a supervised cross entropy loss (Claim 1).
• —Does not cover agent modeling that doesn't involve comparing predicted actions against observed, one-hot encoded actions of other agents (Claim 2).
• —Does not cover neural networks that do not learn opponent or teammate policies, potentially through a latent space (Claim 5, Claim 7).

The clever bit

The clever part is integrating explicit opponent modeling directly into the asynchronous reinforcement learning process. Instead of just reacting to other agents, the AI actively tries to predict their behavior using a supervised loss term, which can significantly improve learning efficiency and the quality of the learned strategies.

Why it matters

This technology is important because it makes training complex artificial intelligence agents much faster and more effective, especially in environments where multiple AIs interact. By explicitly modeling opponents or teammates, the learning agent can develop more sophisticated strategies. The assignee, Royal Bank of Canada, suggests potential applications in areas like financial trading, where predicting the actions of other market participants is critical.

Real-world examples

• 1.AI for complex strategy games (e.g., StarCraft II, Dota 2)
• 2.Autonomous driving systems predicting other vehicle movements
• 3.Financial trading algorithms predicting market participant behavior
• 4.Robotics in collaborative or competitive multi-robot environments
• 5.AI agents for cybersecurity, predicting attacker actions