Android Benchmark App Reports Success Despite Empty Results
@kirklandsign is already working on this.
Since Jun 12, 2025.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🐛 Describe the bug
The example job here: https://github.com/pytorch/executorch/actions/runs/15597347625/job/43931376798
It shows the benchmark job running gemma-3-1b on-device is finished successfully, but when checking the results in the later step (https://github.com/pytorch/executorch/actions/runs/15597347625/job/43932265620#step:8:449), it doesn't report any results from this run. If you downloaded the artifacts from the benchmark job for gemma-3-1b and inspect the results manually, it's empty. That is, there are failures durning inference but the app just silents it and report fake success instead.
For this given model, we know it will crash the runtime due to this bug (https://github.com/pytorch/executorch/issues/11618), so similar to running the llama runner binary, the correct behavior of the app is to fail explicitly and surface the error
Versions
trunk
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.