Azure / Azure/MachineLearningNotebooks

AutoML experiment: get model and metrics for any algorithm (not only for the best one)

Open
#1,885 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

## What I'm trying to do

For an **AutoML Forecasting** experiment, I'd like to compare the performance of the **best model** with the performance of **another model** from the same experiment.

For an AutoML run, I understand how to get the best performing model and its metrics like this:
```
# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...
```

But how can I fetch the job for _any_ model based on the _algorithm name_?
Something like:
```
# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')
```

## So far, I managed to figure out the following:

1. list of algorithms used in the AutoML experiment
```
mlflow_parent_run.data.tags['pipeline_id_000']
# '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'
```
However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.

2. "internal" job names for the child runs

The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance `upbeat_square_abs3942_2` - i.e. the name of the parent run `upbeat_square_abs3942` followed by **underscore plus a number** (`_2`in this example).

But Azure ML Studio displays names like (no `upbeat_square_abs3942_2` to be found):
image
So this code works:
```
child_run = mlflow_client.get_run('upbeat_square_abs3942_2')
```
but using a name shown in the screenshot above throws an exception, e.g.
```
child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
```

### Question

How can I obtain the model and metrics for any algorithm used in the experiment?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the MLflow client usage shown in the issue, including get_run, the automl_best_child_run_id tag, and pipeline_id_000. Trace how AutoML child runs are identified and how algorithm names relate to their internal run names. Done means documenting or enabling retrieval of a model and its metrics for any algorithm in the experiment, not only the best run.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, jupyter-notebook, python
Domain
api, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.