bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed

Implement the ML Flow experiment tracker

Open
#54 9 comments 0 reactions 0 assignees View on GitHub
🌍 Carbon
Dominant language
Python
Stars
1.4k
Forks
226
PR merge metrics
No merged PRs in 30d

Description

**Motivation**. As @sashavor suggested, the carbon footprint working group needs an experiment tracker to properly follow all runs being done. An experiment tracker could also be more broadly interesting to centralise in one place all experiments being done.

**Proposed solution**. Following a discussion with @thomwolf, the carbon WG has identified [**MLFlow**](https://mlflow.org/) as a promising open-source option: it supports having a dedicated server, and can interface with TensorBoard logs (which we already produce). This [blog post](https://medium.com/analytics-vidhya/tensorflow-model-tracking-with-mlflow-e9de29c8e542) shows how it integrates with Tensorflow. There is also [documentation](https://www.mlflow.org/docs/latest/python_api/mlflow.pytorch.html) on how to interface with PyTorch models.

**Implementation**. It's not quite clear how nicely will MLFlow play with Megatron/DeepSpeed+the limited networking on Jean Zay. The goal here is to first build a proof-of-concept showing MLFlow can integrate in our codebase and phone back to the centralised server from Jean Zay. We can then consider the finer details of reporting all metrics of interest to us.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.