About voice interaction

The Voice Interaction Service API provides an abstraction over different potential voice control apps. Implementations can be developed following the guidelines described in App development. The content in this integration guide describes how to integrate these apps into a specific Android Automotive OS (AAOS) system image.

Terminology

These terms are used through this guide:

  • Assist data. When a voice interaction session is started, the system is able to capture views and screenshots, and pass this information to the session. Apps can expose additional information by implementing Activity#onProvideAssistData() and Activity#onProvideAssistContent().
  • Push-to-talk (PTT). Physical voice control button, usually located in the steering wheel.
  • RecognitionService (RS). Voice recognition service used by apps through the SpeechRecognizer API. VIAs must include both the VoiceInteractionService and the RecognitionService.
  • Tap-to-talk (TTT). Software voice control button, usually included as part of the system UI). In Android this is also referred to as Assist Gesture.
  • VoiceInteractionService. Lightweight system service implemented by the VIA developer. The selected service is bound from system service on boot, and is always running.
  • VoiceInteractionSession (VIS). This class encapsulates the user interaction business logic. It is responsible for presenting the user with status of the voice interaction, handling VoiceInteractor requests and receiving assist and screenshot data.
  • VoiceInteractionSessionService (VSS). A service, part of a VIA, responsible for handling a voice interaction session. This service is bound from Android's system service during a voice interaction with a user. All business logic of this session is implemented in the VoiceSession class. This service is only guaranteed to stay alive during a single user voice session.
  • Voice Interaction App (VIA). Android app designed to serve as a voice control (referred to as assistant). These apps can be identified by including a VoiceInteractionService in their manifest. Only one of these apps can be selected as default at a time in the system. Only the default app will be maintained alive (bound from a system service), and will be the receiver of Push-To-Talk (PTT) or Tap-To-Talk (TTT) events.

Responsibilities

This table describes the responsibilities of each party.

Car Manufacturers (OEMs) AOSP App Developers
  • Build a compatible infotainment system with AAOS.
  • Implement audio input and output, optionally including DSP hotword detection support.
  • Grant system-privileged permissions for the voice interaction services.
  • Respect VoiceInteractionService requirements regarding access to app's settings screens.
  • Define and evolve VoiceInteractionService and related APIs.
  • Provide API documentation, sample code and other support material to VIA developers.
  • Provide UX guidance with requirements and recommendations.
  • Implement VoiceInteractionService API, RecognitionService API and NotificationListenerService API (see detailed description at App development).
  • Provide a customizable UI that can be adjusted by OEMs to match each car design system.

UX requirements

OEMs have the ultimate responsibility of providing a good user experience to customers. OEMs must ensure that the all pre-installed voice interaction services fulfill the requirements described in Preloaded Assistants: UX Guidance.

Core assistant experience

An automotive Voice Interaction Application (VIA) performs the following actions:

  • [MUST] Respond to system-handled voice interaction triggers (PTT, TTT).
  • [MUST] Display a visual representation of their progress (for example, listening, processing, and fulfilling).
  • [MUST] Use voice or sounds to indicate understanding and completion of user requests.
  • [MUST] Serve as a speech recognizer for other apps (see the SpeechRecognizer API).
  • [SHOULD] Respond to a hotword trigger.
  • [MAY] Display a settings activity where users can configure this VIA (for example, permissions, hotword configuration, and sign-in).
  • [MAY] Handle assist data (Intent#ACTION_ASSIST)
  • [MAY] Support voice interaction from Keyguard (lock screen).

Components

At a high level, a voice interaction app interacts with these actors:

Voice interaction actors

Figure 1. Voice interaction actors

Details:

  • VoiceInteractionManagerService. This system service is responsible for managing the default VIA, and exposing its functionality to the rest of the system.
  • RecognitionService. This service exposes speech recognition capabilities to other apps in the system.
  • SoundTrigger. Implements hotword management and it's available to VIAs through the AlwaysOnHotwordDetector.
  • MediaRecorder. Provides access to audio input for both hotword detection (when using CPU) and speech recognition.
  • PhoneWindowManager/CarInputService. These services are responsible (among other things) for handling key events, routing PTT to the VIA, by means of the VoiceInteractionManagerService.
  • User. The user interacts with a VIA by means of Triggers (PTT, TTT, Hotword) or the Voice Plate UI.
  • CarService, Notifications, Media, Telephony, ContactsProvider, and so on. Services and apps used by the VoiceInteractionSession to fulfill the user's commands.

Automotive-specific concepts

AAOS diverges from Android in the following aspects:

  • Besides normal Assistant functionalities, AAOS VIAs can control vehicle functions (for example, HVAC, seats, and interior lights). These functionalities can be integrated using the CarPropertyManager API (see more at Read a vehicle property) provided OEMs configure access correctly as described in Privileged permission allowlisting.
  • Customization and consistency are more relevant in Automotive than in any other form factor. See Customization to read more about implementing these guidelines.