# How Voice Assistants Know When You've Finished Talking > This patent describes a method for speech recognition systems to accurately detect the end of a spoken phrase by considering multiple interpretations of the audio and how much silence each interpretation suggests. - **Patent:** US 10121471 - **Original title:** Language model speech endpointing - **Owner:** Amazon Technologies - **Granted:** 2018 - **Status:** Active - **Times cited:** 10 - **Field:** software, telecommunications, consumer_electronics, ai_ml ## What it does This patent details a computer-implemented method for determining when a speaker has finished talking during automatic speech recognition (ASR) processing. It works by receiving audio data, like from a mobile device microphone, and then performing ASR to generate a "plurality of hypotheses" (Claim 1). Each hypothesis is a different guess about what was said. For each guess, the system figures out its probability and how many "non-speech audio frames" (silence or background noise) immediately came before a potential end point. It then calculates a "respective score" for each hypothesis by multiplying its probability by a factor related to its non-speech frames. These scores are summed to create a "cumulative score." If this cumulative score goes above a certain "first threshold," the system decides that the speaker has likely finished speaking at that point. For example, if you say "Hey Alexa, what's the weather?" the system constantly checks if the weighted sum of non-speech from all its guesses indicates you've stopped talking after "weather." ## What it does NOT cover - Does not cover endpointing methods that rely solely on a single best guess (hypothesis) of what was said, rather than considering multiple possibilities. - Does not cover systems that only detect silence or a drop in audio volume without considering the linguistic context or probabilities of different interpretations. - Does not cover endpointing based on a fixed duration of silence after detected speech, without weighting by hypothesis probability. - Does not cover endpointing methods that do not use a "plurality of hypotheses" from an ASR decoder. - Does not cover systems that determine an endpoint based on a user pressing a button to indicate the end of speech. ## The clever bit The cleverness lies in not just looking for silence, but in considering multiple possible interpretations (hypotheses) of the audio simultaneously. By weighting the amount of non-speech each hypothesis suggests by its probability, the system gets a more robust and intelligent estimate of when speech truly ends, even amidst pauses or background noise. ## Real-world examples 1. Amazon Alexa voice assistant 2. Google Assistant 3. Apple Siri 4. Microsoft Cortana 5. Voice control in smart home devices 6. Transcription services using ASR ## Why it matters Accurate speech endpointing is crucial for voice-controlled devices. If a system cuts off too early, it misses part of the command. If it waits too long, the user experiences frustrating delays. This patent helps voice assistants like Amazon's Alexa or Google Assistant respond quickly and correctly, making interactions feel more natural and efficient. It addresses a fundamental challenge in making voice interfaces reliable for everyday use. ## Frequently asked questions ### What does How Voice Assistants Know When You've Finished Talking cover? This patent describes a method for speech recognition systems to accurately detect the end of a spoken phrase by considering multiple interpretations of the audio and how much silence each interpretation suggests. ### Who owns patent US 10121471? Amazon Technologies owns this patent, granted in 2018. ### When does this patent expire? This patent is expected to expire on June 29, 2035, when the invention enters the public domain. ### What is patent US 10121471 cited by? This patent has been cited by 10 later patents that build on its ideas. ### What problem does this patent solve? Accurate speech endpointing is crucial for voice-controlled devices. If a system cuts off too early, it misses part of the command. If it waits too long, the user experiences frustrating delays. This patent helps voice assistants like Amazon's Alexa or Google Assistant respond quickly and correctly, making interactions feel more natural and efficient. It addresses a fundamental challenge in making voice interfaces reliable for everyday use. ### What does this patent NOT cover? Does not cover endpointing methods that rely solely on a single best guess (hypothesis) of what was said, rather than considering multiple possibilities. **Full plain-English explainer:** https://patentbrief.org/patent/us/10121471/language-model-speech-endpointing **Original patent:** https://patents.google.com/patent/US10121471 --- _Source: PatentBrief — https://patentbrief.org. Patent facts are from public records; the plain-English explanation is PatentBrief's._ ## Related patents Semantically similar inventions in the PatentBrief corpus: - [How Voice Assistants Distinguish Speakers After a Wakeword](https://patentbrief.org/patent/us/10923111/speech-detection-and-speech-recognition) — This patent describes a method for voice assistants to better understand who is speaking after a 'wakeword' by comparing voice characteristics to improve speech recognition accuracy, especially in multi-speaker environments. - [How Voice Assistants Listen for More Commands Without a Wake Word](https://patentbrief.org/patent/us/10847149/speech-based-attention-span-for-voice-user-interface) — This patent describes how a voice assistant can keep listening for follow-up commands after an initial request, without needing you to say its wake word again, based on the type of command you just gave. - [How Smart Speakers Know You're Talking to Them After a Command](https://patentbrief.org/patent/us/11361763/detecting-system-directed-speech) — This patent describes how a smart speaker system can tell if follow-up speech is meant for it, even without a "wake word," by analyzing voice activity and partial speech recognition results using an AI model. - [How Voice Assistants Recognize Who is Talking](https://patentbrief.org/patent/us/11514901/anchored-speech-detection-and-speech-recognition) — This patent describes a system that uses a recorded voice sample to identify if new speech comes from the same person, allowing voice assistants to only respond to specific users. - [How AI Predicts Who Will Speak Next in a Conversation](https://patentbrief.org/patent/us/11645473/palm-pathways-language-model) — IBM's patent describes a system that uses neural networks to analyze speech patterns and intentions to predict which person will talk next in a conversation.