{
  "patent_number": "US 10963313",
  "country": "US",
  "title": "How Computer Systems Learn to Manage Themselves and Fix Their Goals",
  "original_title": "Automated reinforcement-learning-based application manager that learns and improves a reward function",
  "summary": "This patent describes an automated system that uses reinforcement learning to manage computer applications and infrastructure, uniquely learning and adjusting its own goal-setting \"reward function\" based on how well it aligns with human expectations.",
  "what_it_does": "The patent describes an \"automated reinforcement-learning-based application manager\" (Claim 1) that oversees a \"computing environment,\" which can include applications, distributed systems, or even single computers. This manager has a set of actions it can perform. It uses an \"iterative control process\" (Claim 1) that repeatedly picks an action based on the current state of the system, performs it, and then gets a \"reward\" (a score) for that action. It uses these rewards to learn how to make better decisions, aiming for an \"optimal or near-optimal control policy\" (Claim 1). The system's key feature is that if its \"reward function\" (which defines what's good or bad) starts to differ from what human managers actually want, the system automatically creates a \"new reward function\" using its past experiences, called \"stored state/action trajectories\" (Claim 1). For example, if the system is supposed to prioritize low latency but starts making choices that increase it, it can detect this mismatch and adjust its internal scoring system to better reflect the human goal of low latency. The system can also incorporate human input directly, either by taking a human-specified action or by proposing an action to a human for approval (Claims 5-7).",
  "what_it_does_not_cover": [
    "Does not cover systems that use a fixed reward function that is never updated or improved based on observed deviations from desired behavior.",
    "Does not cover management systems that rely solely on human input without an automated reinforcement learning process for action selection.",
    "Does not cover reinforcement learning systems that learn a control policy but do not also generate or improve the reward function itself.",
    "Does not cover systems that manage non-computing environments, such as physical robots in a factory, as it specifies \"computing environment\" (Claim 1).",
    "Does not cover systems that generate a new reward function without using \"stored state/action trajectories\" (Claim 1)."
  ],
  "filed": "2019-07-22",
  "granted": "2021-03-30",
  "expires": "2039-07-22",
  "status": "active",
  "holder": "VMware",
  "holder_url": "https://patentbrief.org/company/vmware",
  "inventors": [
    {
      "name": "Nicholas Mark Grant Stephen",
      "url": "https://patentbrief.org/inventor/nicholas-mark-grant-stephen"
    },
    {
      "name": "Dev Nag",
      "url": "https://patentbrief.org/inventor/dev-nag"
    },
    {
      "name": "Dongni Wang",
      "url": "https://patentbrief.org/inventor/dongni-wang"
    },
    {
      "name": "Yanislav Yankov",
      "url": "https://patentbrief.org/inventor/yanislav-yankov"
    },
    {
      "name": "Gregory T. Burk",
      "url": "https://patentbrief.org/inventor/gregory-t-burk"
    }
  ],
  "times_cited": 3,
  "tags": [
    "software",
    "ai_ml",
    "telecommunications",
    "consumer_electronics",
    "semiconductors"
  ],
  "abstract": "The current document is directed to automated reinforcement-learning-based application managers that learn and improve the reward function that steers reinforcement-learning-based systems towards optimal or near-optimal policies. Initially, when the automated reinforcement-learning-based application manager is first installed and launched, the automated reinforcement-learning-based application manager may rely on human-application-manager action inputs and resulting state/action trajectories to accumulate sufficient information to generate an initial reward function. During subsequent operation, when it is determined that the automated reinforcement-learning-based application manager is no longer following a policy consistent with the type of management desired by human application managers, the automated reinforcement-learning-based application manager may use accumulated trajectories to improve the reward function.",
  "url": "https://patentbrief.org/patent/us/10963313/automated-reinforcement-learning-based-application-manager-that-learns-and-impro",
  "markdown_url": "https://patentbrief.org/patent/us/10963313/automated-reinforcement-learning-based-application-manager-that-learns-and-impro/md",
  "google_patents_url": "https://patents.google.com/patent/US10963313",
  "relatedPatents": [
    {
      "patentNumber": "11238372",
      "countryCode": "US",
      "title": "How AI Learns to Manage Computer Applications Using Digital Simulators",
      "url": "https://patentbrief.org/patent/us/11238372/simulator-training-for-automated-reinforcement-learning-based-application-manage"
    },
    {
      "patentNumber": "12001320",
      "countryCode": "US",
      "title": "How AI Learns to Test Software in the Cloud",
      "url": "https://patentbrief.org/patent/us/12001320/cloud-architecture-for-reinforcement-learning"
    },
    {
      "patentNumber": "11429762",
      "countryCode": "US",
      "title": "How Computers Train AI Models Using Separate Virtual Simulations",
      "url": "https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models"
    },
    {
      "patentNumber": "9754221",
      "countryCode": "US",
      "title": "A Computer Chip for Training Many AI Agents at Once",
      "url": "https://patentbrief.org/patent/us/9754221/processor-for-implementing-reinforcement-learning-operations"
    },
    {
      "patentNumber": "11836577",
      "countryCode": "US",
      "title": "Training Robot AI Models Faster Using Smart Simulations",
      "url": "https://patentbrief.org/patent/us/11836577/reinforcement-learning-model-training-through-simulation"
    }
  ]
}