Input Method Editor Support

Updates made to these display-specific areas are provided below:

Android 10 supports software keyboard for apps running on a non-default display.

Apps running on a non-default display

In terms of which display shows the software keyboard of the Input Method Editor (IME), there are two different modes. The software keyboard is shown on the:

  • Same display on which the focused app appears.
  • Default display while the focused app is running on a non-default display.

The system determines which mode to use based on the settings of the display on which the focused app appears. For more details, see:

  • DisplayWindowSettings#shouldShowImeLocked()
  • DisplayWindowSettings#setShouldShowImeLocked()

Figure 1. IME software keyboard as it appears on secondary display, including target app

The system uses a single IME, but can shift between displays to follow user focus. Android 10 automatically expects all first- and third-party IMEs to revise the layout and resize according to the new display size when created.

If there's an active connection on display A, and an input field requests input focus on display B, then the following flow occurs:

  1. A new input connection comes from the input field on display B.
  2. InputMethodManagerService checks if the connection should be approved.
  3. A display is selected for the IME. If display B supports showing the IME and is allowed to show it, then B is used. Otherwise, the primary device display is selected.
  4. If the selected display is not from display A, then the connection is re-established. InputMethodService is destroyed and then created again.

Security restriction

The system won't show an IME on virtual displays that aren't owned by the system. This is due to a security concern that a malicious app could create a virtual display with enabled System Decorations Support and read user-sensitive information from the surface, such as typing predictions and custom backgrounds.

Implementation

In Android 9 (and lower), the IME was only available on the default screen, as described in On-Screen Input methods. In Android 10 (and higher), a user can switch between different input text fields on different displays by switching focus, and the IME window moves to the secondary displays.

The implementation in WindowManager tracks the input method window (the IME window where the soft keyboard is drawn) and the input method target (the window where the IME input goes) to manage the IME state.

For InputMethodManagerService (IMMS), no other built-in mechanism can propagate the display change to InputMethodService (IMS) and reconfigure the keyboard layout at runtime when moving focus to another display.

To achieve the IME window switch between displays, Android 10 implements the following:

  • The IME and input target window are now tracked per display in DisplayContent#mInputMethodWindow and DisplayContent#mInputMethodTarget, so that the WindowManager (WM) can manage the IME focus state independently of each display.
  • On the IMMS side, when an app client's focus request from the external display is received through ViewRootImpl#handleWindowFocusChanged -> InputMethodManager#onPostWindowFocus -> IMMS#startInputOrWindowGainedFocus, it first unbinds the current input method service and then rebinds the service to reattach the new IME window token for the external display in onServiceConnected().
  • On the IMS side, after the IMS#attachToken is received, the following flow occurs:
    • ContextImpl#updateDisplay is called to update the service context's display in InputMethodService#attachToken(). This calls ViewGroup#addView() to revise the layout of the keyboard and adapt to the target display checking the current context.
    • After DisplayContent#setInputMethodWindowLocked() is called, the implementation sends process-level display configuration changes using the WindowProcessController to IME process to override resources and display metrics.
    • The InputMethodService client gets the correct configuration with the correct display metrics after onConfigurationChanged() and the ViewGroup#addView() call to reinitialize the input view.

Multi-session Input Method Editor support

Device implementations with multiple displays expected to be simultaneously used by multiple users to provide appropriate input sources can be configured to simultaneously display multiple input method editors (IME), at most one per display. The following two figures show a sample multi-session IME on two displays:

Figure 2. Sample multi-session IME

Figure 3. Sample multi-session IME

Support for Per-display focus is a prerequisite for this feature. If not, this feature can't be enabled. Due to security restrictions, the per-display focus limitation restricted this feature to a small subset of devices.

In Android 10, support for multi-session IMEs is implemented with separate system services with a different set of APIs and reduced functionality. Multi-session IME isn't compatible with existing IMEs. Either multi-session or single-session service can be used, but not both.

It isn't possible to use existing Android IMEs built on top of the InputMethodService class because the assumption that a single IME client can be focused at the same time was made before Android IME APIs were introduced in Android 1.5 and many public APIs in InputMethodService have already relied heavily on that assumption. However, updating the InputMethodService class to support a multi-client scenario is challenging because:

  1. Doing so would introduce an unacceptable amount of complexity into InputMethodService, which is already hard to maintain.
  2. IME developers still need to update their implementation to be able to support parallel requests from multiple focused IME clients, which may require a nontrivial redesign on their side (such as input decoder and typing history database).
  3. Actual use cases for multi-IME clients are expected to evolve rapidly, so the new protocol isn't stable and isn't ready to be exposed as public APIs.

As with single-session (regular) IME, control over showing IME on individual displays is performed using DisplayWindowSettings.

There is a sample multi-session IME located at development/samples/MultiClientInputMethod.

To test multi-session IME:

  1. Set config_perDisplayFocusEnabled to true.
  2. Run these commands:
    1. $ make -j MultiClientInputMethod
    2. $ adb install -r $OUT/system/priv-app/MultiClientInputMethod/MultiClientInputMethod.apk
    3. $ adb root
    4. $ adb shell setprop persist.debug.multi_client_ime \
      com.example.android.multiclientinputmethod/.MultiClientInputMethod
    5. $ adb reboot
  3. Try multiple text input scenarios.

Implementation

See MultiClientInputMethodManagerService for implementation details.