Voice Assistant Tap-to-Read

Android Automotive considers voice to be a crucial component in the enablement of drive-safe interactions and one of the safest ways for users to interact with the Android Automotive OS while driving. As a result, the existing Android Voice Assistant APIs (including VoiceInteractionSession) have been expanded to enable voice assistants to perform tasks for users that can be difficult to accomplish while driving.

Tap-to-Read enables voice assistants to read and reply to text messages on behalf of that user, when the user interacts with message notifications. To provide this functionality, you can integrate a voice assistant with CarVoiceInteractionSession.

Tap-to-read notification

In Automotive, notifications posted to the Notification Center identified as INBOX or INBOX_IN_GROUP notifications (for example, SMS messages), will include a Play action button. This button allows the user to have the notification be read aloud by the selected VIA and to optionally reply by voice.

Figure 1. Tap-to-Read notification

Integrate with CarVoiceInteractionSession

1. Support VoiceInteractions

Applications that provide car voice interaction services must integrate with the existing Android Voice Interactions (with the exception of VoiceInteractionSession). While all other components in the Voice Interaction API remain the same as implemented on mobile devices, CarVoiceInteractionSession (described below) replaces VoiceInteractionSession. For more information, see these articles:

2. Implement CarVoiceInteractionSession

CarVoiceInteractionSession exposes APIs that can be used to enable voice assistants to read aloud and then reply to text messages on behalf of the user.

When comparing CarVoiceInteractionSession and VoiceInteractionSession, the key difference is that CarVoiceInteractionSession passes in the action in onShow so the voice assistant can detect the context of the user's request as soon as CarVoiceInteractionSession starts a session.

CarVoiceInteractionSession VoiceInteractionSession
onShow takes these three parameters:
  • args
  • showFlags
  • actions
onShow takes these two parameters:
  • args
  • showFlags

Changes in Android 10

Starting with Android 10, the platform calls VoiceInteractionService.onGetSupportedVoiceActions to detect which actions are supported. The voice assistant overrides and implements VoiceInteractionService.onGetSupportedVoiceActions, as shown below:

public class MyInteractionService extends VoiceInteractionService {
    private static final List SUPPORTED_VOICE_ACTIONS = Arrays.asList(
        CarVoiceInteractionSession.VOICE_ACTION_READ_NOTIFICATION);

    @Override
    public Set onGetSupportedVoiceActions(@NonNull Set voiceActions) {
       Set result = new HashSet<>(voiceActions);
       result.retainAll(SUPPORTED_VOICE_ACTIONS);
       return result;
   }
}

Valid actions are described in the table below. For details about each action, see Sequence Diagrams below.

Action Expected Payload Expected Voice Interaction action
VOICE_ACTION_READ_NOTIFICATION Read aloud messages to the user and then fire the Mark as Read pending intent back when the messages are read successfully. Optionally, prompt the user for a reply.
VOICE_ACTION_REPLY_NOTIFICATION Parcelable with key.
KEY_NOTIFICATION that maps to a StatusBarNotification.
Requires android.permission.BIND_NOTIFICATION_LISTENER_SERVICE
Prompt the user to state the reply message, input the reply message into the RemoteInputReply of the pending intent, and then fire the pending intent.
VOICE_ACTION_HANDLE_EXCEPTION String with key.
KEY_EXCEPTION that maps to an ExceptionValue (described below).
KEY_FALLBACK_ASSISTANT_ENABLED that maps to a Boolean value. If the value is true, the Fallback Assistant that can handle the user's request has been disabled.
The expected action to be taken for the exception will be defined in the documentation for the exception.

ExceptionValue

EXCEPTION_NOTIFICATION_LISTENER_PERMISSIONS_MISSING indicates to the voice assistant that it is missing the Manifest.permission.BIND_NOTIFICATION_LISTENER_SERVICE permission and to get this permission from the user.

3. Request Notification Listener Permission

If the default voice assistant does not have Notification Listener Permissions, the platform's FallbackAssistant (if enabled by the car manufacturer) may read the message aloud before the voice assistant is notifed to request the permission. To determine if the FallbackAssistant is enabled and has, therefore, read the message, the voice assistant should check the KEY_FALLBACK_ASSISTANT_ENABLED Boolean value in the payload.

The platform recommends the voice assistant add rate-limiting logic as to the number of times this permission is requested. Doing so respects the user who does not want to grant the voice assistant this permission and prefers the FallbackAssistant to read aloud the text messages. Prompting this user for permission each time the user presses Play on the message notification could lead to a negative user experience. The platform does not impose rate-limits on behalf of the voice assistant.

When requesting the Notification Listener permission, the voice assistant should use CarUxRestrictionsManager to determine if a user is parked or is driving. If the user is driving, the voice assistant displays a notification that provides instructions on how to grant the permission. Doing so helps (and reminds) the user to grant the permission when it is safer.

4. Work with StatusBarNotification

StatusBarNotifications passed in with the Read and Reply voice actions are always in a car-compatible messaging notification as described in Notify users of messages. While some notifications may not have the Reply Pending intent, they all have Mark as Read pending intents.

To streamline interactions with notifications, use NotificationPayloadHandler, which provides methods to extract messages from the notification and write the reply messages to the appropriate pending intent of the notification. After the voice assistant reads the message, the voice assistant must fire the Mark as Read intent.

5. Satisfy Tap-to-Read Pre-Conditions

Only the VoiceInteractionSession of the default voice assistant is notified when a user triggers the voice action to read and reply to messages. As mentioned above, this default voice assistant must also have the Notification Listener permission.

Sequence Diagrams

These figures display the logic flows of CarVoiceInteractionSession actions.

VOICE_ACTION_READ_NOTIFICATION

Figure 2. Sequence diagram for VOICE_ACTION_READ_NOTIFICATION

In the case of Figure 3 below, the application of rate limits on permission requests is recommended.

VOICE_ACTION_REPLY_NOTIFICATION

Figure 3. Sequence diagram for VOICE_ACTION_REPLY_NOTIFICATION

VOICE_ACTION_HANDLE_EXCEPTION

Figure 4. Sequence diagram for VOICE_ACTION_HANDLE_EXCEPTION

Tips and Tricks

Reading the Name of an Application

If your voice assistant would like to read aloud the messaging application's name during the message readout (i.e. "Sam from Hangouts said..."), you should create a function like that shown below to ensure you are reading out the correct name.

@Nullable
String getMessageApplicationName(Context context, StatusBarNotification statusBarNotification) {
    ApplicationInfo info = getApplicationInfo(context, statusBarNotification.getPackageName());
    if (info == null) return null;

    Notification notification = statusBarNotification.getNotification();

    // Sometimes system packages will post on behalf of other applications, so check this
    // field for a system application notification.
    if (isSystemApp(info)
            && notification.extras.containsKey(Notification.EXTRA_SUBSTITUTE_APP_NAME)) {
        return notification.extras.getString(Notification.EXTRA_SUBSTITUTE_APP_NAME);
    } else {
        PackageManager pm = context.getPackageManager();
        return String.valueOf(pm.getApplicationLabel(info));
    }
}

@Nullable
ApplicationInfo getApplicationInfo(Context context, String packageName) {
    final PackageManager pm = context.getPackageManager();
    ApplicationInfo info;
    try {
        info = pm.getApplicationInfo(packageName, 0);
    } catch (PackageManager.NameNotFoundException e) {
        return null;
    }
    return info;
}

boolean isSystemApp(ApplicationInfo info) {
    return (info.flags & ApplicationInfo.FLAG_SYSTEM) != 0;
}