Add accessibility API to interact with other apps

Nobody has claimed this yet.

Feature description

Android API gives the ability to retrieve current structured UI content (where are texts and what are they, where are buttons...) and interact (click, type characters...) with the screen content.

Personal motivation related to Internet-less Wi-Fi required wireless debugging workaround:

This feature would unlock many use cases, notably my open-source voice assistant Benjamin_Loison/Voice_assistant which currently leverages wireless debugging to use adb exec-out uiautomator dump. The issue with wireless debugging is that, despite not using Internet, it requires a Wi-Fi for Termux to connect to the device wireless debugging itself, I planned modifying AOSP to remove this limitation, but I suspect that it is a feature, not a bug in Google's mind and they would not be open to somehow allow proceeding without Wi-Fi. Restricting such possibility to custom ROMs or now on Android versions looks too restrictive for people to enjoy it now.

Personal notes: Benjamin-Loison/android/issues/28

Reference implementation

Have you checked if the feature is accessible through the Android API? Yes, see:

https://developer.android.com/reference/android/view/accessibility/package-summary

Do you know of other open-source apps that has a similar feature as the one you want? (Provide links)

Closed-source apps:

Tasker and AutoInput are closed-source, but leverage this accessibility API to notably retrieve structured screen content. Automate is able to dump the structured screen content as XML.
I just achieved clicking with https://github.com/Nain57/Smart-AutoClicker, but not retrieving structured screen content.

RustDesk is open-source and let the ability through accessibility to click.

I plan to propose a pull request adding commands to leverage accessibility Android API, as my voice assistant is Termux based.

Start with Android's accessibility package summary and the existing Termux API command structure; the issue names no repository files or tests. Define the command surface for retrieving structured UI content and interacting with other apps, then verify that the resulting commands work from Termux.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment