Android: Accessibility: make a fullscreen element so that TalkBack's AI can describe the screen inside games
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 14.1k
- Forks
- 2.2k
- Avg merge
- 7h 35m
- Merged PRs (30d)
- 51
Description
First and foremost consider this:
- Only RetroArch bugs should be filed here. Not core bugs or game bugs
- This is not a forum or a help section, this is strictly developer oriented
Description
In Retroarch's GUI, there are no elements that a screen reader or accessibility service can access. This is fine, since Retroarch speaks menus and dialogs from itself using TalkBack. But, once I'm in a game, it can be helpful to use a screen reader's access to ML models or AI to describe, for example, what's around the character in a game, which menu option is highlighted, stuff like that. So, having just a fullscreen View, or element that TalkBack can latch onto just in order to describe the screen would be helpful. Right now, if an app doesn't have any elements, TalkBack cannot describe the screen, since there is nothing onscreen for it to describe. It shouldn't have to be anything fancy, even a blank "page" or "view" should work, just accessibility title it "retroarch" or something so users aren't confused.
On iOS, we can get around this in VoiceOver by using screen recognition, which uses ML to grab the screen and describe images as a matter of how it works.
Expected behavior
There should be an element covering the screen which TalkBack can detect.
Actual behavior
The screen has no elements, so TalkBack has nothing to describe.
Steps to reproduce the bug
- Turn on TalkBack, and open Retroarch.
- Tap once with three fingers to open the TalkBack menu.
- Scroll to, and activate, Describe Image. If it isn't there, that means it can't detect anything to describe.
TalkBack says "image cannot be described."
Bisect Results
Feature request
Version/Commit
You can find this information under Information/System Information
- RetroArch: 1.19.1
Environment information
- OS: Android 15
- Compiler: [In case you are running local builds]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue on Android 15 with TalkBack enabled, following the listed steps and checking Describe Image. Done means the in-game screen exposes a fullscreen accessible element that TalkBack can detect and use for screen description.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android
- Domain
- accessibility, mobile
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100