element-hq / element-hq/element-android
Consider migrating integration tests from emulator on GHA to another tool (eg Firebase Test Lab)
- Dominant language
- Kotlin
- Stars
- 3.7k
- Forks
- 917
- PR merge metrics
- No merged PRs in 30d
Description
This has a few benefits:
- we still have performance problems and unexpected emulator failures when running in GHA
- we are using (valuable/expensive) macos runners in order to get the emulator to function
- Nice videos of the UI tests failing
- Access to other tests (eg, firebase has robo tests that walk the app generally looking for failures and show screenshots)
There are downsides:
- We can no-longer test against http://10.0.2.2:8080/ as a hardcoded endpoint - we would need to move this selection of endpoint into a option somewhere and default to 10.0.2.2 on developer machines, but use a deployed synapse server for the CI usecase - though a single static one would be fine, I think.
- We would need to pay for these tests if we went over a small daily limit (i've been working on a Spark plan with 10 tests per day for free) - this might make them hard to support for all changes to all community runs, for instance.
- It might be a little slower to run as we need to upload the test suite each time we build (but that's only 30s or so)
Contributor guide
Assessment
This issue has not been assessed yet.