NVIDIA / NVIDIA/TensorRT-Edge-LLM
Performance / TTFT & TPS comparison with vLLM for single VLM inference on Jetson Orin 32GB
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Hi, I'm planning to use this tool for VLM inference on a Jetson Orin 32GB platform. Have you conducted any performance testing on this platform? I'm mainly interested in metrics like TTFT and TPS .
Additionally, do you have any comparison data with vLLM for single inferenceon similar hardware?
I'm trying to understand whether, without considering parallel throughput, your tool would be faster than vLLM for my use case.
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or benchmark entry points are named in the issue. Start by locating the project's existing inference or benchmarking entry point, then define single-inference Jetson Orin 32GB runs measuring TTFT and TPS alongside comparable vLLM runs. Done means the test conditions and comparison data are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- embedded-iot, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100