AI-Hypercomputer / AI-Hypercomputer/maxtext

Support nsys profiler upload in all cases

Aberta
#911 1 comentário 0 reações 1 responsável Reivindicada por @hengtaoguo Ver no GitHub
bug good first issue
Linguagem predominante
Python
Estrelas
2.4k
Forks
607
Merge médio
2d 19h
PRs com merge (30d)
158

Descrição

For both `jax.profiler` (`profiler=xplane` in maxtext) and a GPU nsys profiler (`profiler=nsys` in maxtext) we upload the profile to the `base_output_directory` ([source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/pyconfig.py#L317))

Typically this directory is GCS, it can also be local. However for the nsys profiler we hardcode the uploader to use gsutil [source](https://github.com/AI-Hypercomputer/maxtext/blob/0a919c19911ea2d99445e72a59e838f466b962c6/MaxText/profiler.py#L64), which has two problems
1. Output directory may not be GCS, so gsutil is not applicable
2. Hosts may not have gsutil installed, since gsutil is not in requirements.txt

We should modify the nsys profile upload to work in all cases.

Additional context - https://github.com/AI-Hypercomputer/maxtext/pull/909 was added as a temporary fix for 2 - we won't upload the profile when gsutil is missing, so training may continue

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.