triton-inference-server / triton-inference-server/server
Suggestion to reduce RAM consumption
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11k
- Forks
- 1.8k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 28
Description
Is your feature request related to a problem? Please describe.
So I'm trying to use tritonserver in my project. But it uses a lot of RAM for a single model.
- Is this expected behaviour?
- Are there any tricks to reduce RAM consumption?
Describe the solution you'd like
I'd like to collect list of tips and trick that actually helps to reduce memory footprint.
Describe alternatives you've considered
I've rigorously checked documentation and github issues.
Additional context
My setup looks like as follows: EC2 g4dn.large (4 vCPU, 16 RAM, Nvidia T4). While deploying model consumes like 10Gb of RAM.
Should I consider using instances with more RAM?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Triton documentation and GitHub issues referenced by the reporter, then examine the stated EC2 g4dn.large deployment and its roughly 10 GB RAM use. Done would mean a decided, actionable collection of memory-reduction tips and an explanation of whether this consumption is expected or requires a larger instance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, machine-learning
- Domain
- cloud, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100