NVIDIA-NeMo / NVIDIA-NeMo/Gym

[feat] Add anonymous usage telemetry to NeMo Gym

Open
#3,300 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
324

Description

Background

Many NeMo libraries have anonymous usage telemetry (#767 (https://github.com/NVIDIA-NeMo/Evaluator/pull/767), #680 (https://github.com/NVIDIA-NeMo/Evaluator/pull/680)) to understand adoption patterns and guide roadmap decisions to usage patterns.

Add similar usage telemetry for NeMo Gym

What Evaluator built (reference implementation)

  • TelemetryHandler — async batching, retry logic, graceful degradation (telemetry failures never disrupt the actual run)
  • Structured event types (EvaluationTaskEvent, LauncherJobEvent) — task/harness/model/duration/status fields
  • Opt-out by default, not opt-in: enabled unless NEMO_TELEMETRY_ENABLED=false, with an explicit console notice ("Telemetry enabled. Set NEMO_TELEMETRY_ENABLED=false to disable.")
  • Three telemetry levels (0=off, 1=minimal, 2=full) via env var or persistent config (~/.config/nemo-evaluator/config.yaml); model_id redacted at level 1
  • Session-ID propagation across process boundaries (launcher → evaluator) so events correlate across a multi-process run
  • Configurable endpoint (NEMO_TELEMETRY_ENDPOINT), defaults to production, overridable for staging testing

Goal

Bring Gym to parity: emit anonymous usage events for gym eval run / gym env start (and other relevant commands), e.g.g, what benchmark/environment, what agent/harness, what model server type, duration, success/failure. This should follow the same opt-out/graceful-degradation/redaction posture other NeMo libraries established.

Open questions to resolve before implementation

  • Reuse Evaluator's TelemetryHandler/event infrastructure directly (shared dependency) vs. reimplement in Gym — worth checking if Evaluator's telemetry module is meant to be reusable outside Evaluator, or if Gym needs its own given the different process/session model (multiple concurrent servers vs. a single launcher process).
  • What's the Gym-equivalent event shape? Evaluator's EvaluationTaskEvent maps to one harness/model/task; Gym's gym eval run spans agent + model + resources servers — is one event per rollout run enough, or do we need per-server-type breakdown?
  • Should this land as NEMO_GYM_TELEMETRY_* env vars (consistent naming with Gym's existing NEMO_GYM_OTEL_* convention) or NEMO_TELEMETRY_* (shared naming with Evaluator, easier to reason about across both tools if a user runs both)?
  • Confirm the actual data-collection posture with legal/privacy before landing — same review Evaluator presumably went through.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Evaluator telemetry implementations linked in the issue and locating the Gym commands for eval runs and environment starts. Resolve whether to reuse or reimplement the telemetry infrastructure, define the event and configuration model, and confirm the privacy posture. Done means anonymous events cover the relevant Gym workflows with opt-out, redaction, session handling, and graceful degradation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.