{
  "patent_number": "US 9754221",
  "country": "US",
  "title": "A Computer Chip for Training Many AI Agents at Once",
  "original_title": "Processor for implementing reinforcement learning operations",
  "summary": "This patent describes a specialized computer processor designed to quickly train multiple artificial intelligence agents simultaneously using a unique set of instructions, speeding up how AI learns.",
  "what_it_does": "The system uses a \"first processor\" to set up AI \"reinforcement learning agents\" and their \"environments,\" assigning each a unique ID (Claim 1). A \"first memory module\" stores special instructions, part of an \"application-domain specific instruction set (ASI),\" which include these agent or environment IDs as \"operands\" (Claim 1). A \"complex instruction fetch and decode (CISFD) unit\" then decodes these instructions, creating \"threads\" that also carry the IDs (Claim 1). A \"second processor\" with multiple \"cores\" executes these threads in parallel, applying instructions to the correct agents or environments (Claim 1). This second processor determines actions, \"state-value functions,\" \"Q-values,\" and \"reward values\" for the agents. These values are stored in a \"second memory module\" and then used to train a \"neural network\" via a \"neural network data path\" to improve the AI's learning (Claim 1). For example, a system could train hundreds of robotic agents to navigate different simulated environments simultaneously using these specialized instructions.",
  "what_it_does_not_cover": [
    "General-purpose computer processors running reinforcement learning algorithms without specialized instruction sets.",
    "Reinforcement learning systems that do not use agent or environment IDs as operands within their instructions.",
    "Hardware architectures that do not separate the creation of agents/environments from the execution of learning operations into distinct processors.",
    "Reinforcement learning implementations that do not communicate with a neural network via dedicated data paths for approximating reward or state-value functions.",
    "Systems where multiple agents are processed sequentially rather than in parallel using \"Single Instruction Multiple Agents (SIMA)\" type instructions."
  ],
  "filed": "2017-03-09",
  "granted": "2017-09-05",
  "expires": "2037-03-09",
  "status": "active",
  "holder": "Alphaics",
  "holder_url": "https://patentbrief.org/company/alphaics",
  "inventors": [
    {
      "name": "Nagendra Nagaraja",
      "url": "https://patentbrief.org/inventor/nagendra-nagaraja"
    }
  ],
  "times_cited": 53,
  "tags": [
    "ai_ml",
    "semiconductors",
    "consumer_electronics",
    "software",
    "telecommunications"
  ],
  "abstract": "A reinforcement learning processor specifically configured to execute reinforcement learning operations by the way of implementing an application-specific instruction set is envisaged. The application-specific instruction set incorporates ‘Single Instruction Multiple Agents (SIMA)’ instructions. SIMA type instructions are specifically designed to be implemented simultaneously on a plurality of reinforcement learning agents which interact with corresponding reinforcement learning environments. The SIMA type instructions are specifically configured to receive either a reinforcement learning agent ID or a reinforcement learning environment ID as the operand. The reinforcement learning processor uses neural network data paths to communicate with a neural network which in turn uses the actions, state-value functions, Q-values and reward values generated by the reinforcement learning processor to approximate an optimal state-value function as well as an optimal reward function.",
  "url": "https://patentbrief.org/patent/us/9754221/processor-for-implementing-reinforcement-learning-operations",
  "markdown_url": "https://patentbrief.org/patent/us/9754221/processor-for-implementing-reinforcement-learning-operations/md",
  "google_patents_url": "https://patents.google.com/patent/US9754221",
  "relatedPatents": []
}