{
  "patent_number": "US 11238372",
  "country": "US",
  "title": "How AI Learns to Manage Computer Applications Using Digital Simulators",
  "original_title": "Simulator-training for automated reinforcement-learning-based application-managers",
  "summary": "This patent describes a system that creates and trains digital simulators, which then teach other AI programs how to automatically manage and optimize complex computer applications in a controlled virtual environment.",
  "what_it_does": "The patent describes a \"simulation manager\" that generates and trains specialized simulators (Claim 1). These simulators learn how a computer system behaves when an automated AI application manager takes actions. The simulator uses two main machine learning models: one predicts the next \"state\" (condition) of the system given a current state and an action, and another predicts a \"reward\" (outcome) for that state (Claim 3). To train the simulator, it receives real-world data, including actions, current states, and next states (Claim 5). It then repeatedly compares its predictions to this real data, calculating a \"difference metric\" (like a weighted error, Claim 6), and uses this feedback to adjust its internal models to become more accurate. For example, a human expert could use this system to configure a simulator to mimic a cloud server environment, then use that simulator to train an AI to automatically scale resources for a web application based on user traffic.",
  "what_it_does_not_cover": [
    "It does not cover the automated reinforcement-learning-based application manager itself, only the system that trains it using simulators.",
    "It does not cover simulators that do not learn both a state-transition function (predicting next state) and a reward function (predicting reward) (Claim 3).",
    "It does not cover training methods that do not use a \"difference metric\" computed from simulator-generated and training-data state transitions to adjust simulator parameters (Claim 5).",
    "It does not cover systems where human experts directly manage the applications, only where they provide input for simulator configuration (Claim 1).",
    "It does not cover simulators that are not trained using data collected from an actual computing environment controlled by an automated reinforcement-learning-based application manager (Claim 5)."
  ],
  "filed": "2019-07-22",
  "granted": "2022-02-01",
  "expires": "2039-07-22",
  "status": "active",
  "holder": "VMware",
  "holder_url": "https://patentbrief.org/company/vmware",
  "inventors": [
    {
      "name": "Nicholas Mark Grant Stephen",
      "url": "https://patentbrief.org/inventor/nicholas-mark-grant-stephen"
    },
    {
      "name": "Dev Nag",
      "url": "https://patentbrief.org/inventor/dev-nag"
    },
    {
      "name": "Dongni Wang",
      "url": "https://patentbrief.org/inventor/dongni-wang"
    },
    {
      "name": "Yanislav Yankov",
      "url": "https://patentbrief.org/inventor/yanislav-yankov"
    },
    {
      "name": "Gregory T. Burk",
      "url": "https://patentbrief.org/inventor/gregory-t-burk"
    }
  ],
  "times_cited": 1,
  "tags": [
    "software",
    "ai_ml",
    "telecommunications",
    "cloud_computing",
    "devops"
  ],
  "abstract": "The current document is directed to methods and systems for simulation-based training of automated reinforcement-learning-based application managers. Simulators are generated from data collected from controlled computing environments controlled and may employ any of a variety of different machine-learning models to learn state-transition and reward models. The current disclosed methods and systems provide facilities for visualizing aspects of the models learned by a simulator and for initializing simulator models using domain information. In addition, the currently disclosed simulators employ weighted differences computed from simulator-generated and training-data state transitions for feedback to the machine-learning models to address various biases and deficiencies of commonly employed difference metrics in the context of training automated reinforcement-learning-based application managers.",
  "url": "https://patentbrief.org/patent/us/11238372/simulator-training-for-automated-reinforcement-learning-based-application-manage",
  "markdown_url": "https://patentbrief.org/patent/us/11238372/simulator-training-for-automated-reinforcement-learning-based-application-manage/md",
  "google_patents_url": "https://patents.google.com/patent/US11238372",
  "relatedPatents": [
    {
      "patentNumber": "11429762",
      "countryCode": "US",
      "title": "How Computers Train AI Models Using Separate Virtual Simulations",
      "url": "https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models"
    },
    {
      "patentNumber": "11836577",
      "countryCode": "US",
      "title": "Training Robot AI Models Faster Using Smart Simulations",
      "url": "https://patentbrief.org/patent/us/11836577/reinforcement-learning-model-training-through-simulation"
    },
    {
      "patentNumber": "11615208",
      "countryCode": "US",
      "title": "How Cloud Systems Automatically Create and Train AI Data Models",
      "url": "https://patentbrief.org/patent/us/11615208/dall-e-text-to-image-generation"
    },
    {
      "patentNumber": "20250371349",
      "countryCode": "US",
      "title": "How AI Learns to Run Faster on Specific Computer Chips",
      "url": "https://patentbrief.org/patent/us/20250371349/methods-and-apparatus-for-hardware-aware-machine-learning-model-training"
    },
    {
      "patentNumber": "11295174",
      "countryCode": "US",
      "title": "How AI Agents Learn Faster by Predicting Opponents in Parallel",
      "url": "https://patentbrief.org/patent/us/11295174/opponent-modeling-with-asynchronous-methods-in-deep-rl"
    }
  ]
}