AI-Hypercomputer / AI-Hypercomputer/maxtext

Support nsys profiler upload in all cases

未关闭
#911 1 条评论 0 个 reaction 已指派 1 人 已被 @hengtaoguo 认领 在 GitHub 查看
bug good first issue
主要语言
Python
星标
2.4k
派生
607
平均合并
2 天 19 小时
30 天内合并 PR
158

描述

For both `jax.profiler` (`profiler=xplane` in maxtext) and a GPU nsys profiler (`profiler=nsys` in maxtext) we upload the profile to the `base_output_directory` ([source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/pyconfig.py#L317))

Typically this directory is GCS, it can also be local. However for the nsys profiler we hardcode the uploader to use gsutil [source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/profiler.py#L64), which has two problems
1. Output directory may not be GCS, so gsutil is not applicable
2. Hosts may not have gsutil installed, since gsutil is not in requirements.txt

We should modify the nsys profile upload to work in all cases.

Additional context - https://github.com/AI-Hypercomputer/maxtext/pull/909 was added as a temporary fix for 2 - we won't upload the profile when gsutil is missing, so training may continue

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。