NVIDIA / NVIDIA/TensorRT-Model-Connect
[Platform] Enhance TVM-FFI Bridge and open up kernel slots for flashinfer
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 254
- Forks
- 58
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 235
Description
Summary
Enhance the TVM-FFI bridge to support provider-based execution, with FlashInfer implemented as one optional provider rather than integrated directly into Model Connect.
The bridge should expose a stable provider interface, perform explicit capability checks, and preserve the existing TensorRT Model Connect path as the fallback when no compatible provider is available.
Tasks
- Define the TVM-FFI provider interface and lifecycle.
- Identify the initial operations, models, platforms, and data types to support.
- Implement provider discovery, registration, and capability negotiation through the TVM-FFI bridge.
- Add FlashInfer as the first provider using the bridge interface, without adding a direct FlashInfer dependency to Model Connect.
- Define dependency, ABI, version, and compatibility requirements for providers.
- Implement provider selection based on operation, model, platform, precision, layout, and runtime capabilities.
- Add a safe fallback to the existing TensorRT Model Connect implementation when a provider is unavailable or unsupported.
- Normalize inputs, outputs, errors, and runtime metadata across providers.
- Validate numerical and output-quality parity against the existing implementation.
- Benchmark latency, throughput, startup time, and memory usage.
- Add unit, integration, and qualification coverage for the bridge and FlashInfer provider.
- Document how additional providers can be implemented and registered.
- Document the supported FlashInfer provider configurations and known limitations.
Acceptance criteria
The enhanced TVM-FFI bridge supports provider-based execution, and at least one representative model or operation can use FlashInfer through the FlashInfer provider without a direct FlashInfer integration in Model Connect. Provider selection, compatibility requirements, benchmarks, and fallback behavior are documented, and the existing default execution path has no regression.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named in the issue. Start by locating the existing TensorRT Model Connect path and the TVM-FFI boundary, then use the task list and acceptance criteria to define provider lifecycle, capability checks, fallback behavior, and coverage. Done means FlashInfer works through the bridge without direct Model Connect integration and the default path has no regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, backend-api-design, documentation, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100