NVIDIA / NVIDIA/NeMo-Retriever
[FEA]: NimClient re-write and improvements
Open
Nobody has claimed this yet.
feature request
- Dominant language
- Python
- Stars
- 3k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 116
Description
Is this a new feature, an improvement, or a change to existing functionality?
Improvement
How would you describe the priority of this feature request
Significant improvement
Please provide a clear description of problem this feature solves
The NimClient needs attention and cleanup in a number of areas:
- Improved flexibility in configuration, and the ability to wrap non-Triton gRPC clients, specialized HTTP protocol clients (OpenAI query)
- Improved dynamic batching, including direct and heuristic based determination.
- Improved request and issuance concurrency
Describe the feature, and optionally a solution or implementation and any alternatives
See comment above. The exact implementation will require iteration and some architectural changes.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The payload names no files or tests; start by locating the NimClient entry point and reading its configuration, batching, and concurrency paths. Review how it currently creates Triton gRPC clients and specialized HTTP/OpenAI clients. Done is not concretely defined: the issue needs agreed requirements and architecture for client wrapping, batching, and concurrency before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- backend-api-design, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100