How Voice Assistants Know When You've Finished Talking

This patent describes a method for speech recognition systems to accurately detect the end of a spoken phrase by considering multiple interpretations of the audio and how much silence each interpretation suggests.

What it covers

This patent details a computer-implemented method for determining when a speaker has finished talking during automatic speech recognition (ASR) processing. It works by receiving audio data, like from a mobile device microphone, and then performing ASR to generate a "plurality of hypotheses" (Claim 1). Each hypothesis is a different guess about what was said. For each guess, the system figures out its probability and how many "non-speech audio frames" (silence or background noise) immediately came before a potential end point. It then calculates a "respective score" for each hypothesis by multiplying its probability by a factor related to its non-speech frames. These scores are summed to create a "cumulative score." If this cumulative score goes above a certain "first threshold," the system decides that the speaker has likely finished speaking at that point. For example, if you say "Hey Alexa, what's the weather?" the system constantly checks if the weighted sum of non-speech from all its guesses indicates you've stopped talking after "weather."

What it doesn't cover

• —Does not cover endpointing methods that rely solely on a single best guess (hypothesis) of what was said, rather than considering multiple possibilities.
• —Does not cover systems that only detect silence or a drop in audio volume without considering the linguistic context or probabilities of different interpretations.
• —Does not cover endpointing based on a fixed duration of silence after detected speech, without weighting by hypothesis probability.
• —Does not cover endpointing methods that do not use a "plurality of hypotheses" from an ASR decoder.
• —Does not cover systems that determine an endpoint based on a user pressing a button to indicate the end of speech.

The clever bit

The cleverness lies in not just looking for silence, but in considering multiple possible interpretations (hypotheses) of the audio simultaneously. By weighting the amount of non-speech each hypothesis suggests by its probability, the system gets a more robust and intelligent estimate of when speech truly ends, even amidst pauses or background noise.

Why it matters

Accurate speech endpointing is crucial for voice-controlled devices. If a system cuts off too early, it misses part of the command. If it waits too long, the user experiences frustrating delays. This patent helps voice assistants like Amazon's Alexa or Google Assistant respond quickly and correctly, making interactions feel more natural and efficient. It addresses a fundamental challenge in making voice interfaces reliable for everyday use.

Real-world examples

• 1.Amazon Alexa voice assistant
• 2.Google Assistant
• 3.Apple Siri
• 4.Microsoft Cortana
• 5.Voice control in smart home devices
• 6.Transcription services using ASR