Aggregating manual test results
- Dominant language
- TypeScript
- Stars
- 1.6k
- Forks
- 48
- PR merge metrics
- No merged PRs in 30d
Description
When running Flashlight tests via Maestro with `flashlight test ` the results are nicely aggregated into a single result, which makes comparing against different runs easy. Is it possible to do this when manually executing tests? Not sure if it doesn't exist, or I've just missed it in the docs.
We have found that due to the time Maestro spends letting the app go idle our tests aren't quite as accurate as we'd like them to be. So instead we've been manually running tests via `flashlight measure`, running ~5 times, then grabbing the worst result. Is there a way to merge results like what is done when automating measures, but with manual tests? Happy to help with this if it doesn't exist, just point me in the right direction.
Thanks heaps, love the tool it has massively changed our performance measuring game.
Contributor guide
Research direction
Start with the `flashlight test ` and `flashlight measure` entry points, comparing how automated results are aggregated with the output from repeated manual measurements. Done means manual runs can be merged into one comparable result, with the behavior covered by the relevant CLI tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100