NVIDIA / NVIDIA/Megatron-LM

[ENHANCEMENT] Expose TP/PP/DP ranks as environment variables for integration with tracing tools like CUPTI

Open
#1,674 5 comments 0 reactions 1 assignee Assigned to @skyw View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

First, thank you for providing such a powerful and scalable framework for distributed training.

I would like to propose a small but practical feature that would help with system-level profiling using external tracing tools like NVIDIA CUPTI or DCGM.

While torchrun already sets RANK, LOCAL_RANK, and WORLD_SIZE as environment variables, Megatron-LM internally computes parallelism ranks: TP / DP/ PP Rank using APIs like parallel_state.get_tensor_model_parallel_rank()

However, **these ranks are not exposed as environment variables**, which makes it difficult to access them from low level tools such as CUPTI.

Would you consider adding an optionthat sets the following environment variables after initialize_model_parallel()?

like

**os.environ["TP_RANK"] = str(parallel_state.get_tensor_model_parallel_rank())
os.environ["PP_RANK"] = str(parallel_state.get_pipeline_model_parallel_rank())
os.environ["DP_RANK"] = str(parallel_state.get_data_parallel_rank())**

Would you consider accepting a PR for this?
Thank you again for your work and consideration!
Looking forward to your thoughts.

Best Regards.
[Woosung Myung]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.