AI-Hypercomputer / AI-Hypercomputer/maxtext
Support nsys profiler upload in all cases
- Ngôn ngữ chính
- Python
- Star
- 2.4k
- Fork
- 607
- Merge trung bình
- 2 ngày 19 giờ
- Pull request đã merge (30 ngày)
- 158
Mô tả
For both `jax.profiler` (`profiler=xplane` in maxtext) and a GPU nsys profiler (`profiler=nsys` in maxtext) we upload the profile to the `base_output_directory` ([source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/pyconfig.py#L317))
Typically this directory is GCS, it can also be local. However for the nsys profiler we hardcode the uploader to use gsutil [source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/profiler.py#L64), which has two problems
1. Output directory may not be GCS, so gsutil is not applicable
2. Hosts may not have gsutil installed, since gsutil is not in requirements.txt
We should modify the nsys profile upload to work in all cases.
Additional context - https://github.com/AI-Hypercomputer/maxtext/pull/909 was added as a temporary fix for 2 - we won't upload the profile when gsutil is missing, so training may continue
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.