NVIDIA / NVIDIA/TensorRT-Model-Connect

[Platform] Enhance TVM-FFI Bridge and open up kernel slots for flashinfer

Open
#1,098 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Platform
Dominant language
Python
Stars
254
Forks
58
Avg merge
1d 7h
Merged PRs (30d)
235

Description

Summary

Enhance the TVM-FFI bridge to support provider-based execution, with FlashInfer implemented as one optional provider rather than integrated directly into Model Connect.

The bridge should expose a stable provider interface, perform explicit capability checks, and preserve the existing TensorRT Model Connect path as the fallback when no compatible provider is available.

Tasks

  • Define the TVM-FFI provider interface and lifecycle.
  • Identify the initial operations, models, platforms, and data types to support.
  • Implement provider discovery, registration, and capability negotiation through the TVM-FFI bridge.
  • Add FlashInfer as the first provider using the bridge interface, without adding a direct FlashInfer dependency to Model Connect.
  • Define dependency, ABI, version, and compatibility requirements for providers.
  • Implement provider selection based on operation, model, platform, precision, layout, and runtime capabilities.
  • Add a safe fallback to the existing TensorRT Model Connect implementation when a provider is unavailable or unsupported.
  • Normalize inputs, outputs, errors, and runtime metadata across providers.
  • Validate numerical and output-quality parity against the existing implementation.
  • Benchmark latency, throughput, startup time, and memory usage.
  • Add unit, integration, and qualification coverage for the bridge and FlashInfer provider.
  • Document how additional providers can be implemented and registered.
  • Document the supported FlashInfer provider configurations and known limitations.

Acceptance criteria

The enhanced TVM-FFI bridge supports provider-based execution, and at least one representative model or operation can use FlashInfer through the FlashInfer provider without a direct FlashInfer integration in Model Connect. Provider selection, compatibility requirements, benchmarks, and fallback behavior are documented, and the existing default execution path has no regression.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named in the issue. Start by locating the existing TensorRT Model Connect path and the TVM-FFI boundary, then use the task list and acceptance criteria to define provider lifecycle, capability checks, fallback behavior, and coverage. Done means FlashInfer works through the bridge without direct Model Connect integration and the default path has no regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents, backend-api-design, documentation, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.