# How AI Models Understand Language Using 'Attention' > This patent describes a neural network architecture, known as a Transformer, that uses a "self-attention" mechanism to process sequences of information, like words in a sentence, by weighing the importance of different parts of the input. - **Patent:** US 10452978 - **Original title:** Attention-based sequence transduction neural networks - **Owner:** Google LLC - **Granted:** 2019 - **Status:** Active - **Times cited:** 45 - **Field:** ai_ml, software, telecommunications, consumer_electronics ## What it does The system described takes an input sequence, such as a sentence, and converts it into an output sequence, like a translation. It uses an "encoder neural network" (Claim 1) to process the input, which is made up of multiple "encoder subnetworks." Each subnetwork contains an "encoder self-attention sub-layer" (Claim 1). This sub-layer processes each part of the input by applying a self-attention mechanism, which involves determining a "query" from the current input position, and "keys" and "values" from all input positions (Claim 1). These are then used to generate a specific output for that input position, allowing the network to understand how different parts of the sequence relate to each other. For example, when translating a sentence, this mechanism helps the AI determine which words are most relevant to understanding the meaning of a particular word. ## What it does NOT cover - Does not cover neural networks that process sequences without using a self-attention mechanism. - Does not cover attention mechanisms where the "query," "keys," and "values" are not derived from the *same* subnetwork inputs for self-attention. - Does not cover systems that lack either an "encoder neural network" or a "decoder neural network" as part of the overall sequence transduction network (Claim 1). - Does not cover models that do not explicitly determine and use "query," "keys," and "values" from the subnetwork inputs as described in Claim 1. - Does not cover systems where the initial encoder subnetwork inputs are not formed by combining embedded representations with positional embeddings, as described in Claim 2. ## The clever bit The truly novel aspect is the "self-attention mechanism." Instead of processing sequences one step at a time, it allows the model to simultaneously weigh the importance of every other element in a sequence when processing a single element, efficiently capturing long-range dependencies. ## Real-world examples 1. Google Translate 2. ChatGPT (OpenAI) 3. Bard (Google) 4. BERT (Google) 5. DALL-E (OpenAI) 6. Most modern large language models (LLMs) ## Why it matters This patent covers the core ideas behind the Transformer architecture, which fundamentally changed how artificial intelligence processes sequential data, especially in natural language processing (NLP). It enabled the development of large language models (LLMs) like GPT and BERT, significantly improving tasks such as machine translation, text summarization, and question answering. The architecture allowed for more efficient parallel processing of sequences, making training much faster than previous sequential models. ## Frequently asked questions ### What does How AI Models Understand Language Using 'Attention' cover? This patent describes a neural network architecture, known as a Transformer, that uses a "self-attention" mechanism to process sequences of information, like words in a sentence, by weighing the importance of different parts of the input. ### Who owns patent US 10452978? Google LLC owns this patent, granted in 2019. ### When does this patent expire? This patent is expected to expire on June 28, 2038, when the invention enters the public domain. ### What is patent US 10452978 cited by? This patent has been cited by 45 later patents that build on its ideas. ### What problem does this patent solve? This patent covers the core ideas behind the Transformer architecture, which fundamentally changed how artificial intelligence processes sequential data, especially in natural language processing (NLP). It enabled the development of large language models (LLMs) like GPT and BERT, significantly improving tasks such as machine translation, text summarization, and question answering. The architecture allowed for more efficient parallel processing of sequences, making training much faster than previous sequential models. ### What does this patent NOT cover? Does not cover neural networks that process sequences without using a self-attention mechanism. **Full plain-English explainer:** https://patentbrief.org/patent/us/10452978/transformer-attention-mechanism **Original patent:** https://patents.google.com/patent/US10452978 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [How AI Uses Question-Guided Attention to Answer Questions About Images](https://patentbrief.org/patent/us/9965705/systems-and-methods-for-attention-based-configurable-convolutional-neural-networks-abc-cnn-for-visual-question-answering) — A method for AI to answer questions about images by dynamically focusing on relevant parts of the picture based on the specific question asked. - [Adapting AI Models to Fit Device Resources](https://patentbrief.org/patent/us/20220383078/data-processing-method-and-related-device) — This patent describes how a computer system can automatically shrink a large artificial intelligence model, specifically a "transformer" type, to fit the available computing power of a phone or other device. - [Teaching Computers to Understand Document Similarity Using AI](https://patentbrief.org/patent/us/10909459/federated-learning) — This patent describes a way to train a computer program (a neural network) to understand how similar documents are to each other, by showing it examples and teaching it to group similar ones together and separate dissimilar ones. - [How to Shrink Large AI Models Using Knowledge Distillation](https://patentbrief.org/patent/us/10289962/deep-q-networks-dqn) — A method for teaching small, efficient AI models to mimic the complex decision-making patterns of much larger, more powerful neural networks. - [How to Update AI on Small Devices with Slow Internet](https://patentbrief.org/patent/us/20250363357/systems-and-methods-for-deploying-and-updating-neural-networks-at-the-edge-of-a-) — This patent describes a method for efficiently updating artificial intelligence models on small, internet-connected devices, like smart cameras, by sending only the changes, or 'patches,' instead of the entire updated model, which saves bandwidth.