data_processing metric is incorrect
Open
bug
Performance
research
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
We see things like this reported by nemo-rl
========================= Step 2/100 =========================
▶ Preparing batch...
▶ Taking a training step...
📊 Training Results:
• Loss: 1.4154
• Training FLOPS: 2709.39 TFLOPS (2709.39 TFLOPS per rank)
• Training Model Floating Point Utilization: 273.81%
⏱️ Timing:
• Total step time: 158.50s
• policy_training: 157.59s (99.4%)
• data_processing: 0.90s (0.6%)
Yet further analysis reveals that there are ~50 seconds of extra data processing time that's not captured in metrics.
Contributor guide
Assessment
This issue has not been assessed yet.