[Dynamo] managed generation backend cleanup tracking issue
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Tracking issue for loose ends in the managed Dynamo generation backend introduced in #3391.
Items will be checked off or split into separate issues as they land.
Scope: the `policy.generation.backend: dynamo` path — the driver-owned etcd / NATS / Dynamo-frontend
runtime and its Ray-scheduled vLLM worker fleet. Not Kubernetes, DGD, or externally-deployed Dynamo.
Each item has a stable **`LE-n`** identifier so review comments can point at a specific one. Identifiers are never reused or renumbered; a completed item keeps its number.
### Known missing items
- [ ] **[LE-1]** **Multi-node inference (TP/PP spanning nodes) is unsupported — important for large models.**
`FixedDynamoWorkerPool.start()` hard-rejects any engine group whose GPUs land on more than one node
([worker_pool.py#L124-L128](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/worker_pool.py#L124-L128)):
*"A managed Dynamo engine group spans multiple nodes. Multi-node TP/PP is not supported in the fixed-fleet milestone."*
This caps the backend at models whose TP×PP fits inside one node — for the shipped 4-GPU-per-node
topology that is TP≤4. The vLLM backend already supports this: it derives
`needs_cross_node = model_parallel_size > cluster.num_gpus_per_node`, builds a **unified placement group**,
and sets `NCCL_NVLS_ENABLE=0` for the non-colocated cross-node case
([vllm_generation.py#L74-L78](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_generation.py#L74-L78),
[#L178-L221](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_generation.py#L178-L221)).
Two sub-parts: (a) route Dynamo through `init_cluster_placement_groups` so it gets `use_unified_pg`
instead of the catch-all `else` branch at
[grpo.py#L906-L916](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/grpo.py#L906-L916); (b) verify `dynamo.vllm` can be
driven with a cross-node TP group at all. (Flagged during #3391 review)
- [x] ~~**[LE-2]** **Reuse `WeightSynchronizer` instead of a sixth refit mechanism.**
The refit is wired as `isinstance(policy_generation, DynamoGeneration)` branches inside the legacy
path of `refit_policy_generation` ([grpo.py#L2315](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/grpo.py#L2315)), bypassing the
abstraction that #2444, #2608 and #2971 all landed through. `DynamoGeneration` already implements
every method `CollectiveWeightSynchronizer` calls, with matching signatures — see the review thread
for the concrete port. (Flagged during #3391 review)~~
**(edit: asked for in PR #3391 — the wrong shape here is hard to unwind once `dynamo` is spread across the algorithm layer)**
- [ ] **[LE-3]** **AREAL-style cache invalidation (`recompute_kv_cache_after_weight_updates: true`) is unsupported and fails silently.**
`resume_after_refit` calls `policy_generation.invalidate_kv_cache()` from inside `AsyncTrajectoryCollector`,
which holds a *pickled copy* whose `_managed_runtime` is `None`, so the call raises and is swallowed by a
blanket `except Exception` ([trajectory_collector.py#L567-L575](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/async_utils/trajectory_collector.py#L567-L575)).
The run silently falls back to Magistral-style (keep stale caches) — which is what the shipped recipe selects
anyway, so this is latent today. Either carry `refit_workers()` through `__getstate__` so the copy can POST
`flush_cache` itself, or assert at setup that the flag is false for `backend: dynamo`.
(Flagged during #3391 review)
- [ ] **[LE-4]** **Support the single-controller path (`single_controller.py`, the v2 async GRPO loop).**
`backend: dynamo` is currently rejected there — `_generation_max_len` raises
`ValueError: Unknown generation backend 'dynamo'`
([single_controller_utils/setup.py#L237](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/single_controller_utils/setup.py#L237)) — which
fails loud, correctly, but means the v2 loop cannot use Dynamo at all. Note that wiring it requires the
pickled-copy fix above first: unlike `trajectory_collector.py`, the SC loop calls
`self._gen.invalidate_kv_cache()` **unguarded**
([single_controller.py#L615-L616](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/single_controller.py#L615-L616)), so the same
`_managed_runtime is None` failure would hard-crash the run rather than degrade silently.
SC also builds its synchronizer through `create_weight_synchronizer`
([setup.py#L410](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/single_controller_utils/setup.py#L410)), so the WeightSynchronizer
item above is a prerequisite too. (Flagged during #3391 review)
- [ ] **[LE-5]** **Unit-test both cache-invalidation modes.** There is no test that AREAL-style
(`recompute_kv_cache_after_weight_updates: true`) actually invalidates, nor that Magistral-style skips it.
`test_pickle_roundtrip_drops_driver_owned_runtime` asserts `restored._managed_runtime is None` but never calls
`restored.invalidate_kv_cache()`; the invalidation test covers the driver instance, which production never uses.
TRT-LLM has the precedent — [test_trtllm_generation.py#L296-L318](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/tests/unit/models/generation/trtllm/test_trtllm_generation.py#L296-L318)
parametrizes `(in_flight, recompute_kv, expected_drain)` across all three reachable combinations.
(Flagged during #3391 review)
- [ ] **[LE-6]** **Fault tolerance on Slurm — and a way to simulate it.**
The managed runtime adds four driver-side single points of failure no other backend has (etcd,
`nats-server`, the Dynamo frontend, the token-wrapper uvicorn thread). All liveness checks are
**startup-only** — `_wait_for_frontend`/`_start_etcd`/`_wait_for_port`
([managed_runtime.py#L300](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/managed_runtime.py#L300)) never run again,
so a frontend that dies mid-run takes the job with it. Worker-side,
[`FixedDynamoWorkerPool.validate()`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/worker_pool.py#L168-L196)
detects dead workers and reservations before each refit but only raises — no restart, quarantine, or
re-admission. On a 6-node / 240-minute recipe every one of those losses is currently fatal.
#3454 ("FT & R") is building this for the single-controller path (`engine_supervisor.py`,
`fleet_health.py`, `policy_router.py`, `weight_sync/membership.py`, shard quarantine,
`recreate_worker()`) and validates it with fault injection. Two requirements when Dynamo adopts it:
(a) **reuse #3454's config schema** — `async_rl.watchdog.*`, `async_rl.rollout_failure.*`,
`fleet_health.*` — rather than a parallel `dynamo_cfg`-scoped one, so the same knobs mean the same
thing across backends; (b) **make the failure simulation backend-agnostic**, able to target a
`dynamo.vllm` subprocess, the frontend, etcd or NATS — otherwise Dynamo's four extra SPOFs are exactly
the ones never exercised. Killing the frontend is the interesting case; it has no analog elsewhere.
(Flagged during #3391 review)
- [x] ~~**[LE-7]** **Write a usage guide, separate from the design doc.**
[`docs/design-docs/dynamo-integration.md`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/docs/design-docs/dynamo-integration.md) covers the "why"
well, but there is no "how do I run this". Add `docs/guides/dynamo-generation.md` that links back to the
design doc and then goes straight into: support status (vLLM engine only, non-colocated only, single-node
engine groups, async rollouts only), the derived-image prerequisite, the `dynamo_cfg` + `vllm_cfg` config
layout with the honoured/moved/ignored key table, the 2-GPU smoke run then the 6-node recipe, how to verify
the refit landed (`token_mult_prob_error`), and the backend-specific failure modes. Template:
[`checkpoint-engine-refit.md`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/docs/guides/checkpoint-engine-refit.md) for a feature guide,
[`models/qwen/qwen3-5.md`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/docs/guides/models/qwen/qwen3-5.md) for the support-status/recipe shape.
Register in `docs/index.md`. (Flagged during #3391 review)~~
**(edit: asked for in PR #3391)**
- [x] ~~**[LE-8]** **Make ignored `vllm_cfg` keys loud.** 12 of the 22 `VllmSpecificArgs` keys are silently dropped under
`backend: dynamo`. Most are genuinely N/A (in-process `llm.generate()` knobs, the HTTP/ZMQ refit transports,
`async_engine`), and `enable_return_routed_experts` is already guarded by `validate_router_replay_config` —
but `tool_parser_plugin` and `reasoning_parser_plugin` **moved** to `dynamo_cfg.worker_args.*`, so migrating
a working vLLM config silently changes tool/reasoning parsing. Raise for the moved keys, warn for the
genuinely unsupported ones (`skip_tokenizer_init`, `is_mx`), stay silent for N/A. Typing `vllm_cfg` as a
model rather than `dict[str, Any]` is what makes this enumerable. (Flagged during #3391 review)~~
**(edit: asked for in PR #3391 — the guard and the four-line recipe edit land together)**
- [ ] **[LE-9]** **Router replay (MoE routing replay) is unsupported with Dynamo.**
[`validate_router_replay_config`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/megatron/router_replay.py#L50-L58) raises
*"router_replay.enabled requires vLLM generation."* for any non-vLLM backend, so this fails loud today —
no correctness issue. Two follow-ups if it is ever wired: (a) the token wrapper **overwrites**
`nvext.extra_fields` with `["engine_data"]`
([token_wrapper.py#L387](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/token_wrapper.py#L387)) rather than merging,
so Gym's `routed_experts` request would be silently dropped — ai-dynamo ignores unrecognised
`extra_fields` names, so nothing upstream would complain either; (b) the guard is Megatron-only and its
message says "requires vLLM generation" without mentioning that Dynamo *is* vLLM but not `backend: vllm`.
Worth stating in the usage guide's support-status table regardless. (Flagged during #3391 review)
- [x] ~~**[LE-10]** **Ship a recipe on a public model.** The 6-node SWE recipe targets Nemotron Nano v3.5, which is not
public, and `model_name` defaults to a filesystem placeholder (`/path/to/nemotron-nano-v3.5-checkpoint`)
rather than a HuggingFace ID — `custom_jinja_template` then reads the chat template from inside that same
private directory. Combined with the recipe being in `disabled.txt` and having no L1 functional test, nobody
outside the team can execute the 6-node path at all. The repo already standardises on
`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` (10 recipes), which is also MoE so it still exercises expert
parallelism and the packed NCCL refit. The 2-GPU smoke is fine — it uses public `Qwen/Qwen2.5-1.5B`.
(Flagged during #3391 review)~~
**(edit: asked for in PR #3391 — same one-line edit as the `oc.env` cleanup)**
- [ ] **[LE-11]** **Externally-deployed / unmanaged Dynamo.** Explicitly out of scope for this milestone
([dynamo-integration.md#L4](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/docs/design-docs/dynamo-integration.md#L4)). `DynamoGeneration`
constructs `ManagedDynamoRuntime` unconditionally, so this needs a runtime seam.
(Flagged during #3391 review)
- [ ] **[LE-12]** **Engine backends other than vLLM.** The engine is the hardcoded literal `"dynamo.vllm"`
([dynamo_worker.py#L120](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/dynamo_worker.py#L120)); ai-dynamo v1.3.0
also ships [`sglang`](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/components/src/dynamo/sglang)
and [`trtllm`](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/components/src/dynamo/trtllm) workers.
Needs an `engine` field on `DynamoCfg` rather than another string literal. (Flagged during #3391 review)
- [ ] **[LE-13]** **`expert_parallel_size` is accepted and silently ignored.** `validate_managed_vllm_config`
checks positivity only ([arguments.py#L297-L306](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/dynamo/arguments.py#L297-L306)),
and a bare `--enable-expert-parallel` makes vLLM compute EP = DP×TP = TP. The vLLM backend rejects
`ep != tp` outright ([vllm_generation.py#L125-L136](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_generation.py#L125-L136)).
(Flagged during #3391 review)
- [x] ~~**[LE-14]** **Sync (non-async) rollouts.** `_should_use_async_rollouts` returns `True` unconditionally for
dynamo ([grpo.py#L1947](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/algorithms/grpo.py#L1947)), so `DynamoGeneration.generate()` is
unreachable. Tracking in case a sync entrypoint ever needs it. (Flagged during #3391 review)~~
**(edit: dropped — no known need for a sync entrypoint; deleting the unreachable `generate()` is part of LE-17)**
- [ ] **[LE-18]** **Speculative decoding is accepted but never refit.** `vllm_kwargs.speculative_config`
forwards generically to `--speculative-config`, so `dynamo.vllm` starts *with* a drafter — but the drafter
refit path lives in [`VllmInternalWorkerExtension`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_backend.py#L375-L520)
(`_get_drafter_model`, `_maybe_refit_mtp_drafter`), which the managed path has no equivalent of. Draft weights
stay at step 0 for the whole run: rejection sampling keeps outputs correct, so nothing errors — acceptance
rate just decays and throughput falls back toward non-speculative. For MTP the co-trained layer never reaches
the engine. Supporting this needs a drafter-refit route over Dynamo's admin endpoints, alongside the main
weight transfer. A guard is requested in this PR; the feature is not. (Flagged during #3391 review)
- [ ] **[LE-19]** **Low-precision generation (FP8 / MXFP8 / NVFP4) is unsupported.** Both mechanisms the vLLM
backend uses are unavailable out-of-process: [`init_fp8`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_worker.py#L451-L462)
returns `hf_overrides` *and* monkeypatches `Fp8LinearMethod` / `ModelOptMxFp8FusedMoE` inside NeMo RL's own vLLM
process, and NVFP4 swaps the worker class via
[`resolve_generation_worker_cls`](https://github.com/NVIDIA-NeMo/RL/blob/7224c0dd54f030fd4cac3e247e5999fb76b6bdd9/nemo_rl/models/generation/vllm/vllm_generation.py#L196) →
`VllmQuantGenerationWorker`, whereas `FixedDynamoWorkerPool` hardcodes `DynamoVllmWorker`. Today `precision: fp8`
dies on an unrelated `--dtype` argparse error and NVFP4 is silently ignored. Real support means passing
quantization through `dynamo.vllm`'s own CLI rather than patching the process. A guard is requested in this PR;
the feature is not. (Flagged during #3391 review)
- [ ] **[LE-20]** **No arm64 / GB200 coverage — and the exclusion is a build-arg, not a code property.** `L1_Functional_Tests_Dynamo` is explicitly filtered out of the GB200 functional plan ([cicd-main.yml#L227-L231](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/.github/workflows/cicd-main.yml#L227-L231)), and the SWE1 recipe is registered in `nightly.txt` but not `nightly_gb200.txt`, so nothing exercises `backend: dynamo` on arm64 at any test level. The exclusion is currently unavoidable: `build-container` passes `BUILD_DYNAMO=1` ([#L560](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/.github/workflows/cicd-main.yml#L560)) while `build-container-gb200` does not ([#L606-L609](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/.github/workflows/cicd-main.yml#L606-L609)), and `docker/Dockerfile` short-circuits without it ([#L378-L382](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/docker/Dockerfile#L378-L382)), so `/opt/dynamo_venv` does not exist in the arm64 image. The stated reason — *"the managed Dynamo runtime is currently amd64-only"* — is not a property of this repo's install path: [install.sh#L26-L42](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/docker/dynamo/install.sh#L26-L42) resolves `TARGETARCH` / `uname -m` and accepts `arm64` explicitly, and the etcd and nats-server downloads are arch-parameterised ([#L88](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/docker/dynamo/install.sh#L88), [#L96](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/docker/dynamo/install.sh#L96)). As shipped, that arm64 branch is unreachable. The real constraint is whether `ai-dynamo[vllm]==1.3.0.post1` resolves on aarch64 — worth settling, then either passing `BUILD_DYNAMO=1` on the GB200 build and registering the test, or deleting the dead arm64 branch and failing loudly at install time. (Flagged during #3391 review)
### Known cleanup items
- [x] ~~**[LE-15]** **Automated coverage.** The only acceptance recipe ships in `tests/test_suites/disabled.txt`, the
five new unit-test files are collected by no CI shard, and `docker/dynamo/Dockerfile` is never built in
CI. Needs an L1 functional test (the 2-GPU smoke config is the right size) plus shard registration.
(Flagged during #3391 review)~~
**(edit: landed in #3391.** `tests/functional/L1_Functional_Tests_Dynamo.sh` and `tests/functional/grpo_dynamo.sh` are registered in `cicd-main.yml` and green on `f270519`, asserting [`max(data["train/token_mult_prob_error"]) < 1.05`](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/tests/functional/grpo_dynamo.sh#L74-L76) plus a no-leaked-process check. The SWE1 recipe left `disabled.txt` for [`nightly.txt#L226`](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/tests/test_suites/nightly.txt#L226). `docker/dynamo/Dockerfile` no longer exists — the venv is built inside `docker/Dockerfile` behind `BUILD_DYNAMO=1`, which CI passes on the amd64 build ([cicd-main.yml#L560](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/.github/workflows/cicd-main.yml#L560)), and `uv lock --check --directory docker/dynamo` runs in [lockfile-check.yml#L54-L55](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/.github/workflows/lockfile-check.yml#L54-L55). **Residual arm64 / GB200 gap split out as LE-20.)**
- [ ] **[LE-16]** **Port bands.** `system_port_base: 29000` and the recipes' `25000-28000` / `15001-20000` ranges sit
above the 9000 GB200 ephemeral floor that #2380 and #3103 established; `VLLM_PORT` is not forwarded to
the engine subprocess. (Flagged during #3391 review)
- [x] ~~**[LE-17]** **Dead code.** ~225 removable lines: unreachable `generate()`, three TypedDict twins duplicating
their BaseModel counterparts, `_owns_managed_runtime`, a dead `uv` branch, three unreachable guards in
`token_wrapper.py`. (Flagged during #3391 review)~~
**(edit: asked for in PR #3391 — deletions only)**
- [ ] **[LE-21]** **`generation_metrics/*` ships 14 series where the vLLM backend ships 4.** `snapshot()` renames only the first matching source per alias and leaves every other curated name in the dict ([metrics.py#L189-L195](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/nemo_rl/models/generation/dynamo/metrics.py#L189-L195)), so each aliased Dynamo gauge is logged next to its surviving vLLM twin — `inflight_batch_sizes` + `vllm_num_requests_running`, `num_pending_samples` + `vllm_num_requests_waiting`, `kv_cache_usage_perc` + `vllm_kv_cache_usage_perc` — and every key is plotted twice, `per_worker_` and `average_` ([logger.py#L1177](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/nemo_rl/utils/logger.py#L1177), [#L1194](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/nemo_rl/utils/logger.py#L1194)). Two strays alongside that: `"vllm:num_requests_waiting"` also prefix-matches vLLM 0.23.0's `vllm:num_requests_waiting_by_reason`, and `"vllm_gpu_cache_usage_perc"` ([metrics.py#L54](https://github.com/NVIDIA-NeMo/RL/blob/f270519775de172be226b9bae84d783b4f245dc7/nemo_rl/models/generation/dynamo/metrics.py#L54)) became an unreachable alias source when its curated prefix was dropped in `e4c8b79b6`. Separately worth pinning: `kv_cache_usage_perc` resolves to `dynamo_component_gpu_cache_usage_percent` here but to `vllm:kv_cache_usage_perc` on the vLLM backend, and Dynamo's docs disagree on the scale ([prometheus_names.rs#L788-L789](https://github.com/ai-dynamo/dynamo/blob/v1.3.0/lib/runtime/src/metrics/prometheus_names.rs#L788-L789) says 0.0–1.0, [DASHBOARD_METRICS.md#L73](https://github.com/ai-dynamo/dynamo/blob/v1.3.0/dev/observability/grafana_dashboards/DASHBOARD_METRICS.md#L73) says 0–100). Plot noise only; no training impact. (Flagged during #3391 review)
Add anything else here as it comes up.
Contributor guide
Research direction
Choose one unchecked LE item from the tracking list and start with its named entry point, such as worker_pool.py, trajectory_collector.py, single_controller_utils/setup.py, or managed_runtime.py. Read the referenced tests and related implementation first, then run the focused coverage. Done means that item's stated behavior is implemented and verified without expanding into the explicitly out-of-scope paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100