android / android/nowinandroid

[Bug]: Incorrect screenshot tests failure detection on CI

Open
#1,046 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Kotlin
Stars
21.8k
Forks
4.6k
Avg merge
19h 20m
Merged PRs (30d)
2

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is there a StackOverflow question about this issue?

- [X] I have searched StackOverflow

### What happened?

The current implementation for detecting screenshot changes is based on checking if the `verifyRoborazziDemoDebug` task fails.

- https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L45-L48

- https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L58-L62

Unfortunately, executing this task can fail in multiple ways, not always related to actual screenshot changes.
For example, [this PR](https://github.com/SimonMarquis/nowinandroid/pull/16) introduces a compilation error in the test sources, but is still detected as a screenshot test and [tries to re-update the golden images](https://github.com/SimonMarquis/nowinandroid/pull/16/checks#check-step-10).

It could be great to be able to distinguish errors coming from unexpected screenshot changes, and all others.

Some solutions could be:
- Run the compilation tasks for roborazzi before executing `verifyRoborazziDemoDebug`. It seems like Roborazzi does not yet offer such lifecycle tasks, but could probably easily be implemented on top of JVM's `testClasses` or Android's `compileUnitTestSources`.
- Detect the presence of `outputs/roborazzi/*_compare.png` diff images. This one would be trivial to implement.

### Relevant logcat output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Read .github/workflows/Build.yaml around the verifyRoborazziDemoDebug checks and the referenced Roborazzi output paths. Reproduce the failure distinction using the linked compilation-error example and inspect whether outputs/roborazzi/*_compare.png is present. Done means CI identifies unexpected screenshot changes without treating unrelated test-source compilation failures as screenshot changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kotlin
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.