How Computer Systems Learn to Manage Themselves and Fix Their Goals
This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations.
Original patent title: “Automated reinforcement-learning-based application manager that learns and improves a reward function”
This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations. Granted to VMware in 2021 with 23 claims and 3 forward citations, and it is expected to expire in 2039.
Coverage
What does this patent actually cover?
The patent describes an "automated reinforcement-learning-based application manager" (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1) that oversees a "computing environment," which can include applications, distributed systems, or even single computers. This manager has a set of actions it can perform. It uses an "iterative control process" (Claim 1) that repeatedly picks an action based on the current state of the system, performs it, and then gets a "reward" (a score) for that action. It uses these rewards to learn how to make better decisions, aiming for an "optimal or near-optimal control policy" (Claim 1). The system's key feature is that if its "reward function" (which defines what's good or bad) starts to differ from what human managers actually want, the system automatically creates a "new reward function" using its past experiences, called "stored state/action trajectories" (Claim 1). For example, if the system is supposed to prioritize low latency but starts making choices that increase it, it can detect this mismatch and adjust its internal scoring system to better reflect the human goal of low latency. The system can also incorporate human input directly, either by taking a human-specified action or by proposing an action to a human for approval (ClaimsclaimsThe numbered statements at the end of a patent that legally define what the inventor owns.Read more → 5-7).
The gap
What does this patent NOT cover?
- Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior.
- Does not cover management systems that rely solely on human input without an automated reinforcement learning process for action selection.
- Does not cover reinforcement learning systems that learn a control policy but do not also generate or improve the reward function itself.
- Does not cover systems that manage non-computing environments, such as physical robots in a factory, as it specifies "computing environment" (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1).
- Does not cover systems that generate a new reward function without using "stored state/action trajectories" (ClaimclaimA numbered sentence at the end of a patent that legally defines what the inventor owns. The most important section.Read more → 1).
These exclusions are unique to PatentBrief — derived from the actual claim language, not patent-office boilerplate.
Key facts
What made this novel
The truly novel aspect is the system's ability to self-correct its own "reward function" when it detects that its current goals no longer match the desired operational behavior. Instead of humans constantly tweaking the AI's objectives, the system learns from past interactions and human feedback to refine what it considers "good" or "optimal."
The Patent Drawing
Schematic visualization of the patent's claim structure. Hand-drawn diagrams in progress for each landmark patent.
Where you've seen this
Real-world examples
Automated cloud resource allocation systems
Self-optimizing database management
Intelligent network traffic management
DevOps automation platforms
VMware Aria Operations
Why it matters
The bigger picture
This patent is significant for the field of automated IT operations and cloud management. As computing environments become more complex, manual management is increasingly difficult. This technology allows systems to not only learn how to operate efficiently but also to adapt their understanding of "efficiency" or "desired behavior" over time, reducing the need for constant human intervention. It addresses a fundamental challenge in applying AI to real-world systems: ensuring the AI's goals remain aligned with human intent.
Filed
July 22, 2019
Granted
March 30, 2021
Market context
Who's building on this
Companies in this space
VMware, the assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →, continues to develop advanced management solutions for virtualized and cloud environments, integrating AI and machine learning into products like VMware Aria Operations. Other major cloud providers like Amazon Web Services, Microsoft Azure, and Google Cloud are also heavily invested in automated resource management and intelligent operations, often leveraging similar reinforcement learning principles to optimize their infrastructure and services. Startups focused on AIOps (Artificial Intelligence for IT Operations) also build on these concepts.
Market impact
This patent contributes to the ongoing shift towards autonomous operations in IT infrastructure. It enables more resilient and efficient cloud environments by allowing systems to adapt to changing conditions and evolving performance goals without constant human oversight. This capability reduces operational costs, improves service reliability, and accelerates the adoption of complex distributed computing architectures, making advanced IT management more accessible and scalable.
Claim 1 — Plain English
What this patent covers
The patent describes an "automated reinforcement-learning-based application manager" (Claim 1) that oversees a "computing environment," which can include applications, distributed systems, or even single computers. This manager has a set of actions it can perform. It uses an "iterative control process" (Claim 1) that repeatedly picks an action based on the current state of the system, performs it, and then gets a "reward" (a score) for that action. It uses these rewards to learn how to make better decisions, aiming for an "optimal or near-optimal control policy" (Claim 1). The system's key feature is that if its "reward function" (which defines what's good or bad) starts to differ from what human managers actually want, the system automatically creates a "new reward function" using its past experiences, called "stored state/action trajectories" (Claim 1). For example, if the system is supposed to prioritize low latency but starts making choices that increase it, it can detect this mismatch and adjust its internal scoring system to better reflect the human goal of low latency. The system can also incorporate human input directly, either by taking a human-specified action or by proposing an action to a human for approval (Claims 5-7).
The clever bit
The truly novel aspect is the system's ability to self-correct its own "reward function" when it detects that its current goals no longer match the desired operational behavior. Instead of humans constantly tweaking the AI's objectives, the system learns from past interactions and human feedback to refine what it considers "good" or "optimal."
What it does not cover
- Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior.
- Does not cover management systems that rely solely on human input without an automated reinforcement learning process for action selection.
- Does not cover reinforcement learning systems that learn a control policy but do not also generate or improve the reward function itself.
- Does not cover systems that manage non-computing environments, such as physical robots in a factory, as it specifies "computing environment" (Claim 1).
- Does not cover systems that generate a new reward function without using "stored state/action trajectories" (Claim 1).
Patent timeline
Application submitted to the patent office
Application published, typically 18 months after filing
Patent officially issued
Patent enters public domain
PatentBrief Score
Impact Score
Early stage
Citation count
12/40
Early citations
Claim breadth
15/20
Broad claimsclaimsThe numbered statements at the end of a patent that legally define what the inventor owns.Read more →
Recency
10/20
Granted 5–10 years ago
Assignee scale
0/20
Independent or smaller assigneeassigneeThe entity that owns the patent — usually the inventor's employer or a company.Read more →
PatentBrief Impact Score — based on citation count, claim breadth, recency, and assignee scale. Not a legal assessment.
Heuristic Value Estimate
What this patent might be worth
$62K – $200K
Midpoint $125K · 12.7 yr remaining · industry ×1.6
Heuristic only — blends forward/backward citation counts, claim scope, time remaining, litigation history, and CPC-derived industry baseline. Real valuations need a professional appraisal.
Claim text not yet imported for this patent
The original legal language
Original claims
23 claims as filed with the patent office.
Concepts involved
Citations
Patent lineage
Cite this patent
Stephen, N. M. G., Nag, D., Wang, D., Yankov, Y., & Burk, G. T. (2021). How Computer Systems Learn to Manage Themselves and Fix Their Goals (U.S. Patent No. 10,963,313). U.S. Patent and Trademark Office. https://patentbrief.org/patent/us/10963313/automated-reinforcement-learning-based-application-manager-that-learns-and-impro
Auto-generated from the patent record. Double-check author order and the issue date against the official USPTO document before submitting.
Embed
Add this patent to your site
Drop this plain-English patent card into any blog post or article — free, no signup. It always links back to the full breakdown here.
<div data-patentlens-widget data-patent-number="US10963313"></div> <script src="https://patentbrief.org/embed.js" async></script>
Stay in the loop
Get a weekly digest of new patents.
One email per week. No spam. Unsubscribe anytime.
Keep exploring
Related patents you should know
US 4683195 · 1987
How to Make Billions of Copies of a DNA Segment
This patent describes the Polymerase Chain Reaction (PCR), a method to rapidly create many copies of a specific piece of DNA or RNA, enabling its detection and analysis.
Cetus Corp
US 8697359 · 2014
How to Edit Genes in Human Cells Using an Engineered CRISPR System
This patent describes an engineered CRISPR-Cas9 system for precisely cutting DNA in eukaryotic cells to change how genes work, opening the door for gene editing in complex organisms.
Massachusetts Institute of Technology
US 7657849 · 2010
How the iPhone's Slide-to-Unlock Gesture Works
Apple's 2010 patent describes unlocking a device by dragging a specific graphical image across the touchscreen along a predefined path, a gesture that became iconic with the original iPhone.
Apple Inc
US 4733665 · 1988
How Doctors Implant a Permanent Stent Using a Balloon
This patent describes the method for placing a permanent, expandable wire mesh tube inside a blood vessel or other body tube using a balloon-tipped catheter to widen it and keep it open.
Expandable Grafts Partnership
US 4965188 · 1990
How to Make Many Copies of a DNA Piece with Heat
This patent describes the Polymerase Chain Reaction (PCR) method, a technique to make millions of copies of a specific DNA segment using a heat-resistant enzyme and repeated temperature changes.
Cetus Corp
US 4235871 · 1980
How to Encapsulate Active Materials in Lipid Bubbles Efficiently
This patent describes a method for trapping biologically active substances inside tiny, multi-layered fat bubbles called liposomes, using a specific water-in-oil emulsion and gel-forming process to improve how much material gets captured.
Individual
Semantically similar
You might also find these interesting
US 11238372 · 2022 · VMware
How AI Learns to Manage Computer Applications Using Digital Simulators
US 12001320 · 2024 · Microsoft Technology Licensing
How AI Learns to Test Software in the Cloud
US 11429762 · 2022 · Amazon Technologies
How Computers Train AI Models Using Separate Virtual Simulations
US 9754221 · 2017 · Alphaics
A Computer Chip for Training Many AI Agents at Once
More to explore
More in Software & Internet
US 4405829 · 1983 · Massachusetts Institute of Technology
How RSA Public-Key Encryption Keeps Digital Messages Secret
US 6285999 · 2001 · Leland Stanford Junior University
How Websites Get Ranked by Importance
US 5960411 · 1999 · Amazon com Inc
How Amazon's One-Click Ordering Works for Online Purchases
US 7669123 · 2010 · Facebook Inc
Displaying Friends' Activities in a Social Network Feed
New to patents?
Common Questions
Frequently Asked Questions
What does How Computer Systems Learn to Manage Themselves and Fix Their Goals cover?
This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations.
Who owns patent US 10963313?
VMware owns this patent, granted in 2021.
When does this patent expire?
This patent is expected to expire on July 22, 2039, when the invention enters the public domain.
What is patent US 10963313 cited by?
This patent has been cited by 3 later patents that build on its ideas.
What problem does this patent solve?
This patent is significant for the field of automated IT operations and cloud management. As computing environments become more complex, manual management is increasingly difficult. This technology allows systems to not only learn how to operate efficiently but also to adapt their understanding of "efficiency" or "desired behavior" over time, reducing the need for constant human intervention. It addresses a fundamental challenge in applying AI to real-world systems: ensuring the AI's goals remain aligned with human intent.
What does this patent NOT cover?
Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior.
Same assignee
More from VMware
Patent monitoring