NVIDIA / NVIDIA/TensorRT-Model-Connect
[Platform] Add qualified runtime dispatch to Edge-LLM
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 254
- Forks
- 58
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 235
Description
Summary
Allow Model Connect to dispatch supported workloads to Edge-LLM when a qualified Edge-LLM implementation is available, while retaining the existing Model Connect implementation as the fallback.
Scope
- Discover Edge-LLM runtime capabilities.
- Match qualified profiles using model, revision, platform, precision, layout, and runtime versions.
- Adapt artifacts and data layouts when required.
- Translate configuration and inference requests.
- Normalize outputs, errors, and metrics.
- Fall back cleanly when no qualified Edge-LLM profile exists.
- Make the selected runtime visible in logs and
trtmc inspect. - Add unit and end-to-end tests.
Acceptance criteria
A supported model dispatches to Edge-LLM only when its complete profile matches. Unsupported configurations continue through the native Model Connect path without user-visible breakage.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the trtmc inspect entry point and trace the existing native Model Connect dispatch path. Identify how runtime capabilities, qualified profiles, artifacts, requests, outputs, errors, and metrics are represented before scoping the Edge-LLM integration. Done means complete profile matches use Edge-LLM, unsupported configurations retain the native path, and unit and end-to-end tests cover both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100