NVIDIA-NeMo / NVIDIA-NeMo/Curator

Centralize vLLM setup utilities across inference stages

Open
#1,720 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.8k
Forks
327
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Context

The Nemotron-Parse inference stage (nemo_curator/stages/interleaved/pdf/nemotron_parse/inference.py) contains vLLM setup code (engine args, sampling params, GPU configuration) that is useful for any inference stage using vLLM.

Raised in PR #1693 (VibhuJawa): "Do we have a common way across curator to setup vLLM? If not, we should make this a common place/util to start. Given this is probably the scale at which we have tested this the most GPUs, I don't think we should duplicate this logic."

Proposed work

  • Extract vLLM engine initialization, sampling parameter construction, and GPU/TP configuration into a shared util (e.g. nemo_curator/backends/vllm_utils.py or similar)
  • Refactor NemotronParseInferenceStage to use the shared util
  • Document as the canonical starting point for future vLLM-based inference stages

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nemo_curator/stages/interleaved/pdf/nemotron_parse/inference.py and inspect how NemotronParseInferenceStage builds vLLM engine arguments, sampling parameters, and GPU or tensor-parallel configuration. Extract the reusable setup into a shared backend utility, refactor this stage to use it, and document that utility as the canonical starting point for future vLLM inference stages.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.