android / android/nowinandroid
[Bug]: Incorrect screenshot tests failure detection on CI
- Dominant language
- Kotlin
- Stars
- 21.8k
- Forks
- 4.6k
- Avg merge
- 19h 20m
- Merged PRs (30d)
- 2
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Is there a StackOverflow question about this issue?
- [X] I have searched StackOverflow
### What happened?
The current implementation for detecting screenshot changes is based on checking if the `verifyRoborazziDemoDebug` task fails.
- https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L45-L48
- https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L58-L62
Unfortunately, executing this task can fail in multiple ways, not always related to actual screenshot changes.
For example, [this PR](https://github.com/SimonMarquis/nowinandroid/pull/16) introduces a compilation error in the test sources, but is still detected as a screenshot test and [tries to re-update the golden images](https://github.com/SimonMarquis/nowinandroid/pull/16/checks#check-step-10).
It could be great to be able to distinguish errors coming from unexpected screenshot changes, and all others.
Some solutions could be:
- Run the compilation tasks for roborazzi before executing `verifyRoborazziDemoDebug`. It seems like Roborazzi does not yet offer such lifecycle tasks, but could probably easily be implemented on top of JVM's `testClasses` or Android's `compileUnitTestSources`.
- Detect the presence of `outputs/roborazzi/*_compare.png` diff images. This one would be trivial to implement.
### Relevant logcat output
_No response_
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Read .github/workflows/Build.yaml around the verifyRoborazziDemoDebug checks and the referenced Roborazzi output paths. Reproduce the failure distinction using the linked compilation-error example and inspect whether outputs/roborazzi/*_compare.png is present. Done means CI identifies unexpected screenshot changes without treating unrelated test-source compilation failures as screenshot changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, kotlin
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100