NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer

feat(dev): add managed local model serving tool

Open
#252 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

task
Dominant language
Python
Stars
123
Forks
17
Avg merge
4d 2h
Merged PRs (30d)
11

Description

Priority Level

Medium

Task Summary

Add a source-tree developer tool that deploys and manages local models through one vLLM process lifecycle. The tool should let contributors run Anonymizer against pinned generation and detection models on a GPU host or inside the supplied container without introducing a second deployment domain.

Technical Details & Implementation Plan

The implementation should:

  • Compile pinned TOML profiles into normalized plans with task, model, vllm, and local sections.
  • Provide compile, launch, probe, status, and stop commands.
  • Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through vLLM Factory.
  • Preserve process identity with launch receipts, PID start markers, and process-group ownership checks.
  • Probe task-specific capabilities rather than treating a listening port as readiness.
  • Bound termination and cleanup so failed or stale processes cannot hang the tool indefinitely.
  • Run the same compiler and local-process lifecycle on a GPU host and inside the supplied container.
  • Document the deployment workflow and provide representative pinned profiles.

Acceptance criteria:

  • Compilation is pure and produces deterministic, digest-protected plans.
  • Plan digests are documented as accidental corruption detection, not authentication or semantic proof.
  • Generation and GLiNER profiles share one lifecycle implementation.
  • Capability probes validate chat completions or GLiNER dynamic labels, offsets, and scores as appropriate.
  • Status and stop operations reject stale or reused process identities.
  • Cleanup is bounded and reports whether it completed.
  • Tests cover lifecycle invariants and behavior rather than profile inventory tables.
  • Documentation covers both direct GPU-host deployment and the supplied container.
  • A full Anonymizer dataset run succeeds with locally hosted GLiNER and a generation model, and both services clean up successfully.

Out of scope:

  • Public package APIs for deployment.
  • Remote compute orchestration or attachment to existing endpoints.
  • Docker-specific compiler branches.
  • Native Transformers GLiNER serving.
  • Model-cache discovery.
  • Cryptographic authentication of plans.

Implementation is tracked in PR #212.

Dependencies

vLLM Factory integration is pinned by the repository's local-model dependency group.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR #212 and the repository's local-model dependency group, since the implementation is already tracked there. Check how the source-tree tool should expose compile, launch, probe, status, and stop, then verify the acceptance criteria for deterministic plans, capability probes, bounded cleanup, and both GPU-host and supplied-container workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai-infra-agents, cli, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.