# How Computer Systems Learn to Manage Themselves and Fix Their Goals > This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations. - **Patent:** US 10963313 - **Original title:** Automated reinforcement-learning-based application manager that learns and improves a reward function - **Owner:** VMware - **Granted:** 2021 - **Status:** Active - **Times cited:** 3 - **Field:** software, ai_ml, telecommunications, consumer_electronics, semiconductors ## What it does The patent describes an "automated reinforcement-learning-based application manager" (Claim 1) that oversees a "computing environment," which can include applications, distributed systems, or even single computers. This manager has a set of actions it can perform. It uses an "iterative control process" (Claim 1) that repeatedly picks an action based on the current state of the system, performs it, and then gets a "reward" (a score) for that action. It uses these rewards to learn how to make better decisions, aiming for an "optimal or near-optimal control policy" (Claim 1). The system's key feature is that if its "reward function" (which defines what's good or bad) starts to differ from what human managers actually want, the system automatically creates a "new reward function" using its past experiences, called "stored state/action trajectories" (Claim 1). For example, if the system is supposed to prioritize low latency but starts making choices that increase it, it can detect this mismatch and adjust its internal scoring system to better reflect the human goal of low latency. The system can also incorporate human input directly, either by taking a human-specified action or by proposing an action to a human for approval (Claims 5-7). ## What it does NOT cover - Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior. - Does not cover management systems that rely solely on human input without an automated reinforcement learning process for action selection. - Does not cover reinforcement learning systems that learn a control policy but do not also generate or improve the reward function itself. - Does not cover systems that manage non-computing environments, such as physical robots in a factory, as it specifies "computing environment" (Claim 1). - Does not cover systems that generate a new reward function without using "stored state/action trajectories" (Claim 1). ## The clever bit The truly novel aspect is the system's ability to self-correct its own "reward function" when it detects that its current goals no longer match the desired operational behavior. Instead of humans constantly tweaking the AI's objectives, the system learns from past interactions and human feedback to refine what it considers "good" or "optimal." ## Real-world examples 1. Automated cloud resource allocation systems 2. Self-optimizing database management 3. Intelligent network traffic management 4. DevOps automation platforms 5. VMware Aria Operations ## Why it matters This patent is significant for the field of automated IT operations and cloud management. As computing environments become more complex, manual management is increasingly difficult. This technology allows systems to not only learn how to operate efficiently but also to adapt their understanding of "efficiency" or "desired behavior" over time, reducing the need for constant human intervention. It addresses a fundamental challenge in applying AI to real-world systems: ensuring the AI's goals remain aligned with human intent. ## Frequently asked questions ### What does How Computer Systems Learn to Manage Themselves and Fix Their Goals cover? This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations. ### Who owns patent US 10963313? VMware owns this patent, granted in 2021. ### When does this patent expire? This patent is expected to expire on July 22, 2039, when the invention enters the public domain. ### What is patent US 10963313 cited by? This patent has been cited by 3 later patents that build on its ideas. ### What problem does this patent solve? This patent is significant for the field of automated IT operations and cloud management. As computing environments become more complex, manual management is increasingly difficult. This technology allows systems to not only learn how to operate efficiently but also to adapt their understanding of "efficiency" or "desired behavior" over time, reducing the need for constant human intervention. It addresses a fundamental challenge in applying AI to real-world systems: ensuring the AI's goals remain aligned with human intent. ### What does this patent NOT cover? Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior. **Full plain-English explainer:** https://patentbrief.org/patent/us/10963313/automated-reinforcement-learning-based-application-manager-that-learns-and-impro **Original patent:** https://patents.google.com/patent/US10963313 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [How AI Learns to Manage Computer Applications Using Digital Simulators](https://patentbrief.org/patent/us/11238372/simulator-training-for-automated-reinforcement-learning-based-application-manage) — This patent describes a system that creates and trains digital simulators, which then teach other AI programs how to automatically manage and optimize complex computer applications in a controlled virtual environment. - [How AI Learns to Test Software in the Cloud](https://patentbrief.org/patent/us/12001320/cloud-architecture-for-reinforcement-learning) — This patent describes a cloud system where a single AI model learns to test many live software applications at once by trying actions, getting feedback, and improving itself. - [How Computers Train AI Models Using Separate Virtual Simulations](https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models) — This patent describes a system where one virtual computer runs simulations of a system, like a robot, and another virtual computer uses the simulation data to teach an AI model how to make better decisions. - [A Computer Chip for Training Many AI Agents at Once](https://patentbrief.org/patent/us/9754221/processor-for-implementing-reinforcement-learning-operations) — This patent describes a specialized computer processor designed to quickly train multiple artificial intelligence agents simultaneously using a unique set of instructions, speeding up how AI learns. - [Training Robot AI Models Faster Using Smart Simulations](https://patentbrief.org/patent/us/11836577/reinforcement-learning-model-training-through-simulation) — This patent describes a cloud service that helps train artificial intelligence models for robots by running simulations, even suggesting improvements to the AI's learning rules before starting.