bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
Implement the ML Flow experiment tracker
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
**Motivation**. As @sashavor suggested, the carbon footprint working group needs an experiment tracker to properly follow all runs being done. An experiment tracker could also be more broadly interesting to centralise in one place all experiments being done.
**Proposed solution**. Following a discussion with @thomwolf, the carbon WG has identified [**MLFlow**](https://mlflow.org/) as a promising open-source option: it supports having a dedicated server, and can interface with TensorBoard logs (which we already produce). This [blog post](https://medium.com/analytics-vidhya/tensorflow-model-tracking-with-mlflow-e9de29c8e542) shows how it integrates with Tensorflow. There is also [documentation](https://www.mlflow.org/docs/latest/python_api/mlflow.pytorch.html) on how to interface with PyTorch models.
**Implementation**. It's not quite clear how nicely will MLFlow play with Megatron/DeepSpeed+the limited networking on Jean Zay. The goal here is to first build a proof-of-concept showing MLFlow can integrate in our codebase and phone back to the centralised server from Jean Zay. We can then consider the finer details of reporting all metrics of interest to us.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.