NVIDIA / NVIDIA/TensorRT-LLM

[Usage]: Is there any document for using trt-llm to run pipeline parallel inference unevenly on multiple jetson nodes?

Open
#8,037 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question Scale-out
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info

3 jetson Orin, 1 server with 4090
cuda-python:13.0
cuda:12.6
tensor-RT: v0.12.0-jetson

How would you like to use TensorRT-LLM

Is there any document for using trt-llm to run pipeline parallel inference unevenly on multiple jetson nodes?

For example, we may use pipeline parallel within 3 jetson Orin nodes and distribute uneven portions of the model to each node. More specifically, let's say a model has 80 decoder blocks. I want to deploy 50 on the first node, 20 on the second node, and 10 on the third node.

Is this supported by trt-llm (we not use triton)?

Thank you.

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TensorRT-LLM documentation and examples linked in the issue, then inspect the existing pipeline-parallel documentation for Jetson or multi-node coverage. Determine whether uneven decoder-block placement across three nodes is supported without Triton. Done should be a documented answer with any relevant usage guidance or a clear limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.