[ENHANCEMENT] Expose TP/PP/DP ranks as environment variables for integration with tracing tools like CUPTI
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
First, thank you for providing such a powerful and scalable framework for distributed training.
I would like to propose a small but practical feature that would help with system-level profiling using external tracing tools like NVIDIA CUPTI or DCGM.
While torchrun already sets RANK, LOCAL_RANK, and WORLD_SIZE as environment variables, Megatron-LM internally computes parallelism ranks: TP / DP/ PP Rank using APIs like parallel_state.get_tensor_model_parallel_rank()
However, **these ranks are not exposed as environment variables**, which makes it difficult to access them from low level tools such as CUPTI.
Would you consider adding an optionthat sets the following environment variables after initialize_model_parallel()?
like
**os.environ["TP_RANK"] = str(parallel_state.get_tensor_model_parallel_rank())
os.environ["PP_RANK"] = str(parallel_state.get_pipeline_model_parallel_rank())
os.environ["DP_RANK"] = str(parallel_state.get_data_parallel_rank())**
Would you consider accepting a PR for this?
Thank you again for your work and consideration!
Looking forward to your thoughts.
Best Regards.
[Woosung Myung]
Contributor guide
Assessment
This issue has not been assessed yet.