Colocated full-parameter weight sync: CPU fallback costs ~17% of every training step on seccomp-managed containers; request a pidfd_getfd-free GPU transport
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 312
- Avg merge
- 1h 2m
- Merged PRs (30d)
- 2
Description
**Summary:** On containers where `pidfd_getfd` is blocked (e.g. AutoDL managed instances, seccomp filter mode), colocated weight sync must use the `ROLL_WEIGHT_SYNC_USE_CPU` CPU-staging path. For **full-parameter** models this costs a measured **~103 s per step (~17% of the step wall time)**. This is a feature request for a GPU-side transport that does not depend on `pidfd_getfd` (e.g. NCCL broadcast or pinned-memory staging). Related: #484 reports the same class of problem on the Megatron colocated LoRA path with a different root cause (`CUDA driver error` on an accelerator environment).
**Environment:**
- ROLL @ `370cb24c1036ea9145365478fcc40612b2186fc8`
- Actor: Qwen2.5-3B-Instruct, **full-parameter** PPO (agentic GRPO), bf16 weights ≈ **6.2 GB** per sync
- FSDP2 actor training + vLLM actor inference, colocated (train ⊕ infer on GPU 0/1; fixed-role services on GPU 2), 3× RTX 4090
- vLLM 0.10.2, torch 2.8.0+cu128, ray 2.48.0
- AutoDL managed container, `Seccomp: 2` (filter mode) — `pidfd_getfd` unavailable
- `ROLL_WEIGHT_SYNC_USE_CPU: "1"` (inherited through the example config chain)
**Why the CPU path is mandatory here:** `roll/utils/send_recv_utils.py:317-319` already documents this:
```
# Managed AutoDL containers block pidfd_getfd, which makes CUDA IPC
# deserialization fail even for colocated workers. Preserve the default zero-copy
# path unless this explicit portable transport is requested.
```
The opt-in CPU path (`send_recv_utils.py:320-321`, `bucket.detach().to("cpu").contiguous().numpy()` + `MultiprocessingSerializer`, called from `roll/third_party/fsdp2/model_update.py:229`) works correctly — this is a performance request, not a bug report.
**Measured cost (Qwen2.5-3B full-parameter, n=26 training steps):**
| Metric | Mean |
|---|---|
| `time/actor_train/model_update/total` | **103.1 s** |
| `time/step_model_update` (wall) | 117.0 s |
| `time/step_rollout` | 228.8 s |
| `time/step_train` | 215.3 s |
→ weight sync is **~17% of every training step**, and the single largest fixed per-step cost after rollout and the train pass. For LoRA-sized payloads this overhead is negligible (see #484: 169 MB), but it scales linearly with parameter count — at 7B+ full-parameter this would dominate.
**Request:** a colocated weight-sync transport that does not depend on `pidfd_getfd`, e.g.:
1. NCCL broadcast by extending the existing `_setup_broadcast_group` infrastructure in `roll/third_party/fsdp2/model_update.py` (currently separated/partial-overlap only) to fully-colocated mode, or
2. Pinned-memory staging with explicit `cudaMemcpy` (no IPC handles), or
3. Documented guidance on container runtimes that permit the syscalls CUDA IPC needs.
Happy to share the per-step metric dumps, the full step-time breakdown, or help benchmark alternatives — I have a reproducible 3×4090 colocated setup on the affected environment.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with roll/utils/send_recv_utils.py:317-321 and roll/third_party/fsdp2/model_update.py:229, then inspect the existing _setup_broadcast_group infrastructure. Compare a non-pidfd_getfd transport for fully colocated full-parameter sync and benchmark it on the documented 3×4090 setup; done means avoiding mandatory CPU staging without breaking weight updates in the affected container environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100