NVIDIA-NeMo / NVIDIA-NeMo/Gym

[checkpoint] Make CritPt global batching checkpoint-safe

Open
#3,293 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:environment feature
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
324

Description

Parent

Child of #3223 and #3024.

Shared state

resources_servers/critpt/app.py owns process-global pending submissions, execution membership, futures/waiters, submission/batch counters, API-key cursor, and optional disk cache.

Proposed capability

Persist logical pending submissions/IDs, memberships, monotonic counters, key cursor, and committed cache identity. Rebuild runtime futures and locks, remove retired attempts, and avoid resubmitting committed batches.

Do not serialize futures, locks, credentials, waiters, or raw external handles. Capability reporting must distinguish coordinated-global, drain-only, and restart-only semantics from ordinary per-session export/restore.

Acceptance criteria

  • Restore a partially filled batch and a persisted-before-fire batch without duplicate submission or cross-attempt contamination.
  • Prepare accounts for every accepted shared operation and fails closed at its deadline.
  • Process replacement leaves no stale waiter, batch member, metric interval, or actor reference.
  • Duplicate prepare/restore/resume is idempotent or rejected deterministically.
  • Cold restart and failure cleanup are covered.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resources_servers/critpt/app.py and the parent issues #3223 and #3024 to map the process-global state and checkpoint boundaries. Use the acceptance criteria as the completion checklist, especially restore of partial and persisted-before-fire batches, replacement cleanup, idempotent resume, cold restart, and failure cleanup. Do not serialize runtime-only objects such as futures, locks, credentials, waiters, or external handles.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.