software-mansion / software-mansion/react-native-executorch
Add benchmarking infrastructure
Open
Nobody has claimed this yet.
core
documentation
performance
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 96
- Avg merge
- 20h 51m
- Merged PRs (30d)
- 59
Description
Build automated benchmarking infrastructure to measure and track model performance across devices and configurations.
Currently, benchmarking models consumes a lot of time:
- Manual testing on each device
- No standardized benchmark suite
- Results scattered across different runs/people
- No comparison between model configurations (quantization, delegates)
Goals
- Automated benchmark suite
- Standardized metrics across all models
- Historical tracking to catch performance regressions
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points, so first map the existing model execution and device/configuration paths in the repository. Define the benchmark suite, standardized metrics, result storage, and regression comparison needed to satisfy the listed goals; done means repeatable automated measurements with historical tracking across models and configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch, react-native, typescript
- Domain
- machine-learning, mobile-dev, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100