How Computer Systems Learn to Manage Themselves and Fix Their Goals

This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting "reward function" based on how well it aligns with human expectations.

What it covers

The patent describes an "automated reinforcement-learning-based application manager" (Claim 1) that oversees a "computing environment," which can include applications, distributed systems, or even single computers. This manager has a set of actions it can perform. It uses an "iterative control process" (Claim 1) that repeatedly picks an action based on the current state of the system, performs it, and then gets a "reward" (a score) for that action. It uses these rewards to learn how to make better decisions, aiming for an "optimal or near-optimal control policy" (Claim 1). The system's key feature is that if its "reward function" (which defines what's good or bad) starts to differ from what human managers actually want, the system automatically creates a "new reward function" using its past experiences, called "stored state/action trajectories" (Claim 1). For example, if the system is supposed to prioritize low latency but starts making choices that increase it, it can detect this mismatch and adjust its internal scoring system to better reflect the human goal of low latency. The system can also incorporate human input directly, either by taking a human-specified action or by proposing an action to a human for approval (Claims 5-7).

What it doesn't cover

• —Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior.
• —Does not cover management systems that rely solely on human input without an automated reinforcement learning process for action selection.
• —Does not cover reinforcement learning systems that learn a control policy but do not also generate or improve the reward function itself.
• —Does not cover systems that manage non-computing environments, such as physical robots in a factory, as it specifies "computing environment" (Claim 1).
• —Does not cover systems that generate a new reward function without using "stored state/action trajectories" (Claim 1).

The clever bit

The truly novel aspect is the system's ability to self-correct its own "reward function" when it detects that its current goals no longer match the desired operational behavior. Instead of humans constantly tweaking the AI's objectives, the system learns from past interactions and human feedback to refine what it considers "good" or "optimal."

Why it matters

This patent is significant for the field of automated IT operations and cloud management. As computing environments become more complex, manual management is increasingly difficult. This technology allows systems to not only learn how to operate efficiently but also to adapt their understanding of "efficiency" or "desired behavior" over time, reducing the need for constant human intervention. It addresses a fundamental challenge in applying AI to real-world systems: ensuring the AI's goals remain aligned with human intent.

Real-world examples

• 1.Automated cloud resource allocation systems
• 2.Self-optimizing database management
• 3.Intelligent network traffic management
• 4.DevOps automation platforms
• 5.VMware Aria Operations