NVIDIA / NVIDIA/TensorRT-Model-Connect

[Platform] Add qualified runtime dispatch to Edge-LLM

Open
#1,099 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Platform
Dominant language
Python
Stars
254
Forks
58
Avg merge
1d 7h
Merged PRs (30d)
235

Description

Summary

Allow Model Connect to dispatch supported workloads to Edge-LLM when a qualified Edge-LLM implementation is available, while retaining the existing Model Connect implementation as the fallback.

Scope

  • Discover Edge-LLM runtime capabilities.
  • Match qualified profiles using model, revision, platform, precision, layout, and runtime versions.
  • Adapt artifacts and data layouts when required.
  • Translate configuration and inference requests.
  • Normalize outputs, errors, and metrics.
  • Fall back cleanly when no qualified Edge-LLM profile exists.
  • Make the selected runtime visible in logs and trtmc inspect.
  • Add unit and end-to-end tests.

Acceptance criteria

A supported model dispatches to Edge-LLM only when its complete profile matches. Unsupported configurations continue through the native Model Connect path without user-visible breakage.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the trtmc inspect entry point and trace the existing native Model Connect dispatch path. Identify how runtime capabilities, qualified profiles, artifacts, requests, outputs, errors, and metrics are represented before scoping the Edge-LLM integration. Done means complete profile matches use Edge-LLM, unsupported configurations retain the native path, and unit and end-to-end tests cover both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.