NVIDIA / NVIDIA/TensorRT-Model-Connect

[Benchmark] Integrate the ALE benchmark

Open
#1,104 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Platform
Dominant language
Python
Stars
254
Forks
58
Avg merge
1d 7h
Merged PRs (30d)
235

Description

Summary

Add reproducible support for running the ALE benchmark against TensorRT Model Connect implementations.

The benchmark integration must pin the benchmark definition and model inputs precisely enough that contributors can reproduce published quality and performance baselines.

Tasks

  • Confirm the benchmark's full name, authoritative source, version or revision, tasks, datasets, and licensing requirements.
  • Define the supported benchmark tasks, models, checkpoints, and input/output adapters.
  • Pin model revisions, benchmark data revisions, precision, bundle configuration, and runtime versions.
  • Add a reproducible benchmark command that works from a clean documented environment.
  • Define warmup, measurement, synchronization, and result-aggregation procedures.
  • Record accuracy or quality, valid-output rate, latency, throughput, and memory where applicable.
  • Preserve per-sample failure information rather than reporting only aggregate successful results.
  • Compare results with the original framework implementation using aligned inputs, outputs, and measurement boundaries.
  • Publish machine-readable benchmark metadata and results.
  • Add a lightweight CI smoke test that verifies setup, adapter, execution, and result-schema behavior without running the full benchmark.
  • Document full benchmark reproduction steps and known limitations.

Acceptance criteria

A contributor can start from a clean environment, run the documented command, and reproduce a published ALE baseline using a pinned benchmark revision, model revision, and configuration.

The published result includes quality, validity, performance, memory, environment, and comparison metadata sufficient to interpret and reproduce the claim.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by confirming ALE's full name, authoritative source, revision, tasks, datasets, and licensing requirements. Then identify the benchmark, model, adapter, runtime, and result-schema entry points in the repository and define pinned configurations and measurement boundaries. Done means a clean documented environment reproduces the published baseline, emits machine-readable metadata and results, and passes a lightweight CI smoke test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.