Azure / Azure/MachineLearningNotebooks

AutoML experiment: get model and metrics for any algorithm (not only for the best one)

未關閉
#1,885 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Jupyter Notebook
星號
4.4k
分支
2.6k
PR 合併指標
30 天內沒有已合併 PR

描述

## What I'm trying to do

For an **AutoML Forecasting** experiment, I'd like to compare the performance of the **best model** with the performance of **another model** from the same experiment.

For an AutoML run, I understand how to get the best performing model and its metrics like this:
```
# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...
```

But how can I fetch the job for _any_ model based on the _algorithm name_?
Something like:
```
# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')
```

## So far, I managed to figure out the following:

1. list of algorithms used in the AutoML experiment
```
mlflow_parent_run.data.tags['pipeline_id_000']
# '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'
```
However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.

2. "internal" job names for the child runs

The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance `upbeat_square_abs3942_2` - i.e. the name of the parent run `upbeat_square_abs3942` followed by **underscore plus a number** (`_2`in this example).

But Azure ML Studio displays names like (no `upbeat_square_abs3942_2` to be found):
image
So this code works:
```
child_run = mlflow_client.get_run('upbeat_square_abs3942_2')
```
but using a name shown in the screenshot above throws an exception, e.g.
```
child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
```

### Question

How can I obtain the model and metrics for any algorithm used in the experiment?

Thanks!

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 issue 中展示的 MLflow 用戶端用法開始,包括 get_run、automl_best_child_run_id 標籤和 pipeline_id_000。追蹤 AutoML 子執行的識別方式,以及演算法名稱與其內部執行名稱之間的關係。完成的標準是記錄或支援取得實驗中任意演算法的模型及其指標,而不僅僅是最佳執行。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
azure, jupyter-notebook, python
領域
api, machine-learning
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。