{
  "patent_number": "US 11120365",
  "country": "US",
  "title": "How AI Models Learn Complex Tasks by Breaking Them Down",
  "original_title": "For hierarchical decomposition deep reinforcement learning for an artificial intelligence model",
  "summary": "This patent describes a method for training artificial intelligence models to solve big problems by splitting them into smaller, manageable steps and learning each step separately before combining them.",
  "what_it_does": "The patent details an Artificial Intelligence (AI) engine with an instructor module and a learner module (Claim 1). The instructor module uses a \"hierarchical-decomposition reinforcement learning technique\" to break a \"complex task\" into multiple \"individual sub-tasks\" (Claim 1). Each sub-task becomes a \"concept node\" within a \"hierarchical graph\" that forms part of the AI model (Claim 1). The learner module then trains the AI objects corresponding to these individual sub-tasks, often performing this training \"in parallel at the same time\" (Claim 1). Crucially, the AI engine uses specific \"reward functions\" for each individual sub-task and then separate reward functions for the overall \"end solution\" of the complex task (Claim 1). This approach, combining parallel training and focused reward functions, is designed to \"speed up an overall training duration\" compared to training the entire complex task with a single algorithm (Claim 1). For example, teaching a robot to prepare a meal could be decomposed into sub-tasks like \"identify ingredients,\" \"chop vegetables,\" and \"cook food,\" each trained with its own reward, before learning how to integrate them for the complete meal.",
  "what_it_does_not_cover": [
    "Does not cover AI training that uses only a single, monolithic algorithm for a complex task without any decomposition into sub-tasks.",
    "Does not cover AI training methods that do not employ separate reward functions for individual sub-tasks and the overall complex task's solution.",
    "Does not cover AI training where the individual sub-tasks are not represented as \"concept nodes\" within a \"hierarchical graph\" structure.",
    "Does not cover AI training where the individual sub-tasks are not trained, at least initially, in parallel.",
    "Does not cover AI models where user input is not used to automatically partition individual sub-tasks into concept nodes."
  ],
  "filed": "2018-06-14",
  "granted": "2021-09-14",
  "expires": "2038-06-14",
  "status": "active",
  "holder": "Microsoft Technology Licensing",
  "holder_url": "https://patentbrief.org/company/microsoft-technology-licensing",
  "inventors": [
    {
      "name": "Aditya GUDIMELLA",
      "url": "https://patentbrief.org/inventor/aditya-gudimella"
    },
    {
      "name": "Victor SHNAYDER",
      "url": "https://patentbrief.org/inventor/victor-shnayder"
    },
    {
      "name": "Ross Story",
      "url": "https://patentbrief.org/inventor/ross-story"
    },
    {
      "name": "Marcos Campos",
      "url": "https://patentbrief.org/inventor/marcos-campos"
    },
    {
      "name": "Matthew Brown",
      "url": "https://patentbrief.org/inventor/matthew-brown"
    },
    {
      "name": "Ruofan Kong",
      "url": "https://patentbrief.org/inventor/ruofan-kong"
    },
    {
      "name": "Matineh Shaker",
      "url": "https://patentbrief.org/inventor/matineh-shaker"
    }
  ],
  "times_cited": 5,
  "tags": [
    "ai_ml",
    "software",
    "telecommunications",
    "consumer_electronics"
  ],
  "abstract": "Methods and apparatuses that apply a hierarchical-decomposition reinforcement learning technique to train one or more AI objects as concept nodes composed in a hierarchical graph incorporated into an AI model. The individual sub-tasks of a decomposed task may correspond to its own concept node in the hierarchical graph and are initially trained on how to complete their individual sub-task and then trained on how the all of the individual sub-tasks need to interact with each other in the complex task in order to deliver an end solution to the complex task. Next, during the training, using reward functions focused for solving each individual sub-task and then a separate one or more reward functions focused for solving the end solution of the complex task. In addition, where reasonably possible, conducting the training of the AI objects corresponding to the individual sub-tasks in the complex task, in parallel at the same time.",
  "url": "https://patentbrief.org/patent/us/11120365/for-hierarchical-decomposition-deep-reinforcement-learning-for-an-artificial-int",
  "markdown_url": "https://patentbrief.org/patent/us/11120365/for-hierarchical-decomposition-deep-reinforcement-learning-for-an-artificial-int/md",
  "google_patents_url": "https://patents.google.com/patent/US11120365",
  "relatedPatents": [
    {
      "patentNumber": "11429762",
      "countryCode": "US",
      "title": "How Computers Train AI Models Using Separate Virtual Simulations",
      "url": "https://patentbrief.org/patent/us/11429762/simulation-orchestration-for-training-reinforcement-learning-models"
    },
    {
      "patentNumber": "11295174",
      "countryCode": "US",
      "title": "How AI Agents Learn Faster by Predicting Opponents in Parallel",
      "url": "https://patentbrief.org/patent/us/11295174/opponent-modeling-with-asynchronous-methods-in-deep-rl"
    },
    {
      "patentNumber": "20250371349",
      "countryCode": "US",
      "title": "How AI Learns to Run Faster on Specific Computer Chips",
      "url": "https://patentbrief.org/patent/us/20250371349/methods-and-apparatus-for-hardware-aware-machine-learning-model-training"
    },
    {
      "patentNumber": "11170293",
      "countryCode": "US",
      "title": "How AI Systems Learn to Predict and Act Simultaneously",
      "url": "https://patentbrief.org/patent/us/11170293/alphago-policy-and-value-networks"
    },
    {
      "patentNumber": "11836577",
      "countryCode": "US",
      "title": "Training Robot AI Models Faster Using Smart Simulations",
      "url": "https://patentbrief.org/patent/us/11836577/reinforcement-learning-model-training-through-simulation"
    }
  ]
}