How AI Models Understand Language Using 'Attention'

This patent describes a neural network architecture, known as a Transformer, that uses a "self-attention" mechanism to process sequences of information, like words in a sentence, by weighing the importance of different parts of the input.

What it covers

The system described takes an input sequence, such as a sentence, and converts it into an output sequence, like a translation. It uses an "encoder neural network" (Claim 1) to process the input, which is made up of multiple "encoder subnetworks." Each subnetwork contains an "encoder self-attention sub-layer" (Claim 1). This sub-layer processes each part of the input by applying a self-attention mechanism, which involves determining a "query" from the current input position, and "keys" and "values" from all input positions (Claim 1). These are then used to generate a specific output for that input position, allowing the network to understand how different parts of the sequence relate to each other. For example, when translating a sentence, this mechanism helps the AI determine which words are most relevant to understanding the meaning of a particular word.

What it doesn't cover

• —Does not cover neural networks that process sequences without using a self-attention mechanism.
• —Does not cover attention mechanisms where the "query," "keys," and "values" are not derived from the *same* subnetwork inputs for self-attention.
• —Does not cover systems that lack either an "encoder neural network" or a "decoder neural network" as part of the overall sequence transduction network (Claim 1).
• —Does not cover models that do not explicitly determine and use "query," "keys," and "values" from the subnetwork inputs as described in Claim 1.
• —Does not cover systems where the initial encoder subnetwork inputs are not formed by combining embedded representations with positional embeddings, as described in Claim 2.

The clever bit

The truly novel aspect is the "self-attention mechanism." Instead of processing sequences one step at a time, it allows the model to simultaneously weigh the importance of every other element in a sequence when processing a single element, efficiently capturing long-range dependencies.

Why it matters

This patent covers the core ideas behind the Transformer architecture, which fundamentally changed how artificial intelligence processes sequential data, especially in natural language processing (NLP). It enabled the development of large language models (LLMs) like GPT and BERT, significantly improving tasks such as machine translation, text summarization, and question answering. The architecture allowed for more efficient parallel processing of sequences, making training much faster than previous sequential models.

Real-world examples

• 1.Google Translate
• 2.ChatGPT (OpenAI)
• 3.Bard (Google)
• 4.BERT (Google)
• 5.DALL-E (OpenAI)
• 6.Most modern large language models (LLMs)