NVIDIA-NeMo / NVIDIA-NeMo/Gym

[checkpoint] Declare and test LongMT restart-only actor pools

Open
#3,296 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:environment feature
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Parent

Child of #3223 and #3024.

Shared state

resources_servers/longmt_eval/app.py owns a lazily initialized SegALE/Ray evaluator actor pool, initialization flag, lock, and reconstructible caches rather than per-rollout semantic state.

Proposed capability

Advertise restart-only dependency recreation, rebuild and health-check actors before resume, and keep in-flight verification outside a published checkpoint.

Do not serialize futures, locks, credentials, waiters, or raw external handles. Capability reporting must distinguish coordinated-global, drain-only, and restart-only semantics from ordinary per-session export/restore.

Acceptance criteria

  • Equivalent verification succeeds after cold actor recreation; stale actor references and initialization failure surface clearly without claiming export/restore.
  • Prepare accounts for every accepted shared operation and fails closed at its deadline.
  • Process replacement leaves no stale waiter, batch member, metric interval, or actor reference.
  • Duplicate prepare/restore/resume is idempotent or rejected deterministically.
  • Cold restart and failure cleanup are covered.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in resources_servers/longmt_eval/app.py and trace the evaluator actor pool, initialization flag, lock, and reconstructible caches. Read the parent issues #3223 and #3024 for surrounding design, then run the existing evaluation checks before defining coverage for cold actor recreation, deadlines, process replacement, duplicate operations, and failure cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.