How AI Agents Learn to Pick the Best Future Actions
A method for an AI agent to predict which actions will yield the highest rewards by analyzing past experiences and refining its decision-making model.
Original patent title: “Action selection with a reward estimator applied to machine learning”
A method for an AI agent to predict which actions will yield the highest rewards by analyzing past experiences and refining its decision-making model. Granted to Sony Corp in 2019 with 14 claims, and it is expected to expire in 2035.
Coverage
What does this patent actually cover?
This patent describes a system where an AI agent learns from its history to make better decisions. It records 'action history data'—which includes the state the agent was in, the action it took, and the reward it received. The system uses this data to build a 'reward estimator' that predicts how much reward a future action might generate. By comparing these predicted rewards for various possible next steps, the agent selects and executes the action with the highest estimated value. This process allows the agent to continuously improve its performance as it gathers more data.
The gap
What does this patent NOT cover?
- Does not cover general machine learning algorithms that do not specifically use reward estimation based on action history data.
- Does not cover hardware-specific implementations, as the claimsclaimsThe numbered statements at the end of a patent that legally define what the inventor owns.Read more → focus on the logical process performed by a CPU.
- Does not cover reinforcement learning methods that rely solely on trial-and-error without a basis-function-based reward estimator.
These exclusions are unique to PatentBrief — derived from the actual claim language, not patent-office boilerplate.
Key facts
What made this novel
The system uses 'basis functions' to transform raw state and action data into 'feature amount vectors,' which allows the AI to map complex, high-dimensional experiences into a space where it can more easily calculate and predict rewards.
The Patent Drawing
Schematic visualization of the patent's claim structure. Hand-drawn diagrams in progress for each landmark patent.
Where you've seen this
Real-world examples
Autonomous robot navigation in warehouses
Non-player character (NPC) behavior in video games
Automated resource management in cloud computing
Why it matters
The bigger picture
This technology is fundamental to modern autonomous systems, such as robotics and game AI, where an agent must navigate complex environments. By formalizing how an agent evaluates the potential 'reward' of its next move, Sony provides a framework for more efficient decision-making in unpredictable scenarios. It represents a shift toward more structured, data-driven behavior in automated agents.
Filed
June 12, 2015
Granted
May 7, 2019
Market context
Who's building on this
Companies in this space
Sony continues to integrate advanced AI into its robotics and gaming divisions, such as the Aibo robotic dog and AI-driven features in the PlayStation ecosystem. Other major players in autonomous agents, like Boston Dynamics and various industrial automation firms, utilize similar reward-estimation architectures.
Market impact
This patent formalizes a specific approach to reinforcement learning that helps standardize how AI agents prioritize actions. It provides a technical roadmap for developers to build agents that can adapt to new information, which is critical for the growth of the autonomous robotics and intelligent software markets.
Claim 1 — Plain English
What this patent covers
This patent describes a system where an AI agent learns from its history to make better decisions. It records 'action history data'—which includes the state the agent was in, the action it took, and the reward it received. The system uses this data to build a 'reward estimator' that predicts how much reward a future action might generate. By comparing these predicted rewards for various possible next steps, the agent selects and executes the action with the highest estimated value. This process allows the agent to continuously improve its performance as it gathers more data.
The clever bit
The system uses 'basis functions' to transform raw state and action data into 'feature amount vectors,' which allows the AI to map complex, high-dimensional experiences into a space where it can more easily calculate and predict rewards.
What it does not cover
- Does not cover general machine learning algorithms that do not specifically use reward estimation based on action history data.
- Does not cover hardware-specific implementations, as the claims focus on the logical process performed by a CPU.
- Does not cover reinforcement learning methods that rely solely on trial-and-error without a basis-function-based reward estimator.
Patent timeline
Application submitted to the patent office
Application published, typically 18 months after filing
Patent officially issued
Patent enters public domain
PatentBrief Score
Impact Score
Early stage
Citation count
0/40
No citations yet
Claim breadth
9/20
Moderate scope
Recency
10/20
Granted 5–10 years ago
Assignee scale
20/20
Major company or institution
PatentBrief Impact Score — based on citation count, claim breadth, recency, and assignee scale. Not a legal assessment.
Heuristic Value Estimate
What this patent might be worth
$24K – $77K
Midpoint $48K · 8.9 yr remaining · industry ×1.6
Heuristic only — blends forward/backward citation counts, claim scope, time remaining, litigation history, and CPC-derived industry baseline. Real valuations need a professional appraisal.
Claim text not yet imported for this patent
The original legal language
Original claims
14 claims as filed with the patent office.
Concepts involved
Citations
Patent lineage
Cite this patent
Kobayashi, Y. (2019). How AI Agents Learn to Pick the Best Future Actions (U.S. Patent No. 10,282,665). U.S. Patent and Trademark Office. https://patentbrief.org/patent/us/10282665/action-selection-with-a-reward-estimator-applied-to-machine-learning
Auto-generated from the patent record. Double-check author order and the issue date against the official USPTO document before submitting.
Embed
Add this patent to your site
Drop this plain-English patent card into any blog post or article — free, no signup. It always links back to the full breakdown here.
<div data-patentlens-widget data-patent-number="US10282665"></div> <script src="https://patentbrief.org/embed.js" async></script>
Stay in the loop
Get a weekly digest of new patents.
One email per week. No spam. Unsubscribe anytime.
Keep exploring
Related patents you should know
US 4683195 · 1987
How to Make Billions of Copies of a DNA Segment
This patent describes the Polymerase Chain Reaction (PCR), a method to rapidly create many copies of a specific piece of DNA or RNA, enabling its detection and analysis.
Cetus Corp
US 8697359 · 2014
How to Edit Genes in Human Cells Using an Engineered CRISPR System
This patent describes an engineered CRISPR-Cas9 system for precisely cutting DNA in eukaryotic cells to change how genes work, opening the door for gene editing in complex organisms.
Massachusetts Institute of Technology
US 7657849 · 2010
How the iPhone's Slide-to-Unlock Gesture Works
Apple's 2010 patent describes unlocking a device by dragging a specific graphical image across the touchscreen along a predefined path, a gesture that became iconic with the original iPhone.
Apple Inc
US 4733665 · 1988
How Doctors Implant a Permanent Stent Using a Balloon
This patent describes the method for placing a permanent, expandable wire mesh tube inside a blood vessel or other body tube using a balloon-tipped catheter to widen it and keep it open.
Expandable Grafts Partnership
US 4965188 · 1990
How to Make Many Copies of a DNA Piece with Heat
This patent describes the Polymerase Chain Reaction (PCR) method, a technique to make millions of copies of a specific DNA segment using a heat-resistant enzyme and repeated temperature changes.
Cetus Corp
US 4235871 · 1980
How to Encapsulate Active Materials in Lipid Bubbles Efficiently
This patent describes a method for trapping biologically active substances inside tiny, multi-layered fat bubbles called liposomes, using a specific water-in-oil emulsion and gel-forming process to improve how much material gets captured.
Individual
Semantically similar
You might also find these interesting
US 10607134 · 2020
How AI Learns to Control Game Characters Based on Their Surroundings
US 11836577 · 2023 · Amazon Technologies
Training Robot AI Models Faster Using Smart Simulations
US RE46310 · 2017 · Blanding Hovenweep LLC
How Devices Predict Your Next Move Using Gestures and History
US 12423586 · 2025 · D5AI
Making AI Smarter by Focusing on Unsure 'Nodes'
More to explore
More in AI & Machine Learning
US 10452978 · 2019 · Google LLC
How AI Models Understand Language Using 'Attention'
US 6523026 · 2003 · Huntsman International LLC
How Computers Find Hidden Connections Between Different Fields of Knowledge
US 11615208 · 2023 · Capital One Services LLC
How Cloud Systems Automatically Create and Train AI Data Models
US 10402750 · 2019 · Facebook Inc
How Facebook Uses Deep Learning to Predict What You Might Like
New to patents?
Common Questions
Frequently Asked Questions
What does How AI Agents Learn to Pick the Best Future Actions cover?
A method for an AI agent to predict which actions will yield the highest rewards by analyzing past experiences and refining its decision-making model.
Who owns patent US 10282665?
Sony Corp owns this patent, granted in 2019.
When does this patent expire?
This patent is expected to expire on June 12, 2035, when the invention enters the public domain.
What problem does this patent solve?
This technology is fundamental to modern autonomous systems, such as robotics and game AI, where an agent must navigate complex environments. By formalizing how an agent evaluates the potential 'reward' of its next move, Sony provides a framework for more efficient decision-making in unpredictable scenarios. It represents a shift toward more structured, data-driven behavior in automated agents.
What does this patent NOT cover?
Does not cover general machine learning algorithms that do not specifically use reward estimation based on action history data.
Same assignee
More from Sony Corp
Patent monitoring