bamlab / bamlab/flashlight

Aggregating manual test results

Open
#171 2 comments 0 reactions 0 assignees View on GitHub
Package:CLI support
Dominant language
TypeScript
Stars
1.6k
Forks
48
PR merge metrics
No merged PRs in 30d

Description

When running Flashlight tests via Maestro with `flashlight test ` the results are nicely aggregated into a single result, which makes comparing against different runs easy. Is it possible to do this when manually executing tests? Not sure if it doesn't exist, or I've just missed it in the docs.

We have found that due to the time Maestro spends letting the app go idle our tests aren't quite as accurate as we'd like them to be. So instead we've been manually running tests via `flashlight measure`, running ~5 times, then grabbing the worst result. Is there a way to merge results like what is done when automating measures, but with manual tests? Happy to help with this if it doesn't exist, just point me in the right direction.

Thanks heaps, love the tool it has massively changed our performance measuring game.

Contributor guide

Open the contributing guide

Research direction

Start with the `flashlight test ` and `flashlight measure` entry points, comparing how automated results are aggregated with the output from repeated manual measurements. Done means manual runs can be merged into one comparable result, with the behavior covered by the relevant CLI tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.