5% flops utilization during Qwen3-8B SFT
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
I'm SFT'ing Deepseek-R1-Qwen3-8B and I achieve only ~5% flops utilization. This seems abnormally low, so I'm wondering if it's a reporting issue, config issue, or nemo-rl issue.
Repro:
* august 26th: `d168de3f`
* 16 nodes H100
* bf16, seqlen=32K, gbs=512, mbs=1, tp=2, cp=1, sequence_parallel=true. DTensorV1.
* https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B/tree/main
* full config attached below
Things I've tried:
* Applying the fix from https://github.com/NVIDIA-NeMo/RL/pull/1034, actually cuts reported flops utilization in half to 5%
* Removing torch.cuda.empty_cache from https://github.com/NVIDIA-NeMo/RL/issues/1036. MFU 2.5 --> 5%
* shmooing (mbs, tp, cp). Other combos either OOM or are slower
* Tried enabling sequence packing but met https://github.com/NVIDIA-NeMo/RL/issues/1051
* Tried instead using megatron config but got an error from MegatronPolicyWorker
```
File "/opt/ray_venvs/nemo_rl.models.policy.megatron_policy_worker.MegatronPolicyWorker/lib/python3.12/site-packages/transformers/generation/configuration_utils.py", line 1050, in from_pretrained
resolved_config_file = cached_file(
^^^^^^^^^^^^
File "/opt/ray_venvs/nemo_rl.models.policy.megatron_policy_worker.MegatronPolicyWorker/lib/python3.12/site-packages/transformers/utils/hub.py", line 312, in cached_file
file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/ray_venvs/nemo_rl.models.policy.megatron_policy_worker.MegatronPolicyWorker/lib/python3.12/site-packages/transformers/utils/hub.py", line 427, in cached_files
raise OSError(
OSError: /hf_models/DeepSeek-R1-0528-Qwen3-8B does not appear to have a file named generation_config.json. Checkout 'https://huggingface.co//hf_models/DeepSeek-R1-0528-Qwen3-8B/tree/main' for available files.
```
* Tried dtensor v2 by setting `dtensor_cfg._v2=true` but got 1.8% MFU instead of 5%
I've attached configs and logs below.
[output.log](https://github.com/user-attachments/files/22181285/output.log)
[config.yaml](https://github.com/user-attachments/files/22181286/config.yaml)
Contributor guide
Assessment
This issue has not been assessed yet.