libretro / libretro/RetroArch

Android: Accessibility: make a fullscreen element so that TalkBack's AI can describe the screen inside games

Open
#16,968 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
14.1k
Forks
2.2k
Avg merge
7h 35m
Merged PRs (30d)
51

Description

First and foremost consider this:

  • Only RetroArch bugs should be filed here. Not core bugs or game bugs
  • This is not a forum or a help section, this is strictly developer oriented

Description

In Retroarch's GUI, there are no elements that a screen reader or accessibility service can access. This is fine, since Retroarch speaks menus and dialogs from itself using TalkBack. But, once I'm in a game, it can be helpful to use a screen reader's access to ML models or AI to describe, for example, what's around the character in a game, which menu option is highlighted, stuff like that. So, having just a fullscreen View, or element that TalkBack can latch onto just in order to describe the screen would be helpful. Right now, if an app doesn't have any elements, TalkBack cannot describe the screen, since there is nothing onscreen for it to describe. It shouldn't have to be anything fancy, even a blank "page" or "view" should work, just accessibility title it "retroarch" or something so users aren't confused.

On iOS, we can get around this in VoiceOver by using screen recognition, which uses ML to grab the screen and describe images as a matter of how it works.

Expected behavior

There should be an element covering the screen which TalkBack can detect.

Actual behavior

The screen has no elements, so TalkBack has nothing to describe.

Steps to reproduce the bug
  1. Turn on TalkBack, and open Retroarch.
  2. Tap once with three fingers to open the TalkBack menu.
  3. Scroll to, and activate, Describe Image. If it isn't there, that means it can't detect anything to describe.

TalkBack says "image cannot be described."

Bisect Results

Feature request

Version/Commit

You can find this information under Information/System Information

  • RetroArch: 1.19.1
Environment information
  • OS: Android 15
  • Compiler: [In case you are running local builds]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue on Android 15 with TalkBack enabled, following the listed steps and checking Describe Image. Done means the in-game screen exposes a fullscreen accessible element that TalkBack can detect and use for screen description.

Written by the indexing model from the issue text.

Assessment

Tech stack
android
Domain
accessibility, mobile
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.