NVIDIA / NVIDIA/TensorRT-Model-Connect
[Benchmark] Integrate the ALE benchmark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 254
- Forks
- 58
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 235
Description
Summary
Add reproducible support for running the ALE benchmark against TensorRT Model Connect implementations.
The benchmark integration must pin the benchmark definition and model inputs precisely enough that contributors can reproduce published quality and performance baselines.
Tasks
- Confirm the benchmark's full name, authoritative source, version or revision, tasks, datasets, and licensing requirements.
- Define the supported benchmark tasks, models, checkpoints, and input/output adapters.
- Pin model revisions, benchmark data revisions, precision, bundle configuration, and runtime versions.
- Add a reproducible benchmark command that works from a clean documented environment.
- Define warmup, measurement, synchronization, and result-aggregation procedures.
- Record accuracy or quality, valid-output rate, latency, throughput, and memory where applicable.
- Preserve per-sample failure information rather than reporting only aggregate successful results.
- Compare results with the original framework implementation using aligned inputs, outputs, and measurement boundaries.
- Publish machine-readable benchmark metadata and results.
- Add a lightweight CI smoke test that verifies setup, adapter, execution, and result-schema behavior without running the full benchmark.
- Document full benchmark reproduction steps and known limitations.
Acceptance criteria
A contributor can start from a clean environment, run the documented command, and reproduce a published ALE baseline using a pinned benchmark revision, model revision, and configuration.
The published result includes quality, validity, performance, memory, environment, and comparison metadata sufficient to interpret and reproduce the claim.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by confirming ALE's full name, authoritative source, revision, tasks, datasets, and licensing requirements. Then identify the benchmark, model, adapter, runtime, and result-schema entry points in the repository and define pinned configurations and measurement boundaries. Done means a clean documented environment reproduces the published baseline, emits machine-readable metadata and results, and passes a lightweight CI smoke test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100