tensorflow / tensorflow/models
Adding Support for MLFlow Experimentation using Graph Model
@pkulzc is already working on this.
Since Mar 5, 2021.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
This is in refrence to the issue here - https://github.com/mlflow/mlflow/issues/3367
I have been using MLFlow to track object detection experiments, metrics and results. It's data science lifecycle management toolkit.
It works perfectly fine with TensorFlow Sequential and Functional API models, training and evaluation. However, this current object detection repository (TF2.X) uses custom training and evaluation loops and some of the methods are non-traditional which are causing MLFlow code to not function as appropriately and error out.
Changes -
- Adding callbacks options to the training/evaluation function calls.
- Adding return statements from
eager_train_step,eager_eval_loopandexport_inference_graph.
Only way to get the training metrics is Run training in eager mode which makes mlflow's patch functionality capture eval metrics (numpy values) from return.
Probably running training in eager mode is the most concerning as I did see training runs slows down drastically (Eg. 100 steps taking an hour to run versus 10mins). I did see the TF warnings saying they are trying to improve this part of training.
I'm also open to see if there are workarounds for getting this to work without doing any changes.
Let me know how I can be helpful here. I can loop MLFLow folks here if needed.
Libraries Used -
Python -3.8.5
TensorFlow - 2.X
MLFlow - 1.14.X
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.