How Digital Assistants Control Apps and Ask for More Information

This patent describes how a digital assistant on a device can understand what a user wants from a natural language command, find the right app, get a step-by-step guide from another device, and then ask the user for more details on the screen to complete the task with that app.

What it covers

This patent details a system where an electronic device, like a smartphone, receives a natural-language voice command from a user. The device then figures out the user's 'intent' (what they want to do) and identifies a specific software application on the device that can perform that task. Uniquely, the device then receives a 'task flow'—a series of programmed steps—from a *second electronic device* (Claim 1). This task flow guides how the primary device interacts with the identified app. If the app needs more information to complete the task, it sends a request. The device then uses the touch-sensitive display to show a 'query' (a question) to the user, determined by the task flow. After the user provides a 'second user input' (the answer to the query), the device sends this response back to the app to help it finish the task. For example, if you tell your phone, "Order a pizza," the phone identifies a pizza ordering app, gets the steps for ordering from a server, and if the app needs to know the topping, it might display, "What toppings would you like?" on your screen.

What it doesn't cover

• —Does not cover scenarios where the 'task flow' (the step-by-step guide) is not received from a 'second electronic device' but is instead entirely pre-programmed or generated on the primary device itself (Claim 1).
• —Does not cover situations where the digital assistant's 'query' for more information from the user is provided solely through voice, without using a 'touch-sensitive display' (Claim 1).
• —Does not cover applications that are not 'stored on the electronic device' but run entirely in the cloud or on a remote server (Claim 1).
• —Does not cover user inputs that are not 'natural-language user input,' such as direct button presses, gestures, or menu selections (Claim 1).
• —Does not cover systems where the identified software application does not send a 'request' for additional information, meaning the task can be completed without further user interaction (Claim 1).

The clever bit

The clever part is how the system dynamically fetches a 'task flow' from another device to guide interactions with an app, and then uses that flow to generate specific questions for the user on the display when the app needs more details, feeding those answers back to the app.

Why it matters

This patent is significant because it outlines a sophisticated way for digital assistants to interact with and control other applications, especially third-party ones. It allows assistants like Siri to handle more complex, multi-step tasks by dynamically asking the user for missing information. This capability is crucial for making digital assistants truly useful beyond simple, built-in commands, expanding their reach into a vast ecosystem of applications.

Real-world examples

• 1.Apple Siri interacting with third-party apps like Uber or DoorDash
• 2.Google Assistant fulfilling requests with installed apps
• 3.Amazon Alexa routines that involve multiple steps and user confirmation
• 4.Any modern smartphone digital assistant that can control other applications via voice.