Azure / Azure/MachineLearningNotebooks
AutoML experiment: get model and metrics for any algorithm (not only for the best one)
- 主要語言
- Jupyter Notebook
- 星號
- 4.4k
- 分支
- 2.6k
- PR 合併指標
- 30 天內沒有已合併 PR
描述
## What I'm trying to do
For an **AutoML Forecasting** experiment, I'd like to compare the performance of the **best model** with the performance of **another model** from the same experiment.
For an AutoML run, I understand how to get the best performing model and its metrics like this:
```
# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...
```
But how can I fetch the job for _any_ model based on the _algorithm name_?
Something like:
```
# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')
```
## So far, I managed to figure out the following:
1. list of algorithms used in the AutoML experiment
```
mlflow_parent_run.data.tags['pipeline_id_000']
# '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'
```
However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.
2. "internal" job names for the child runs
The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance `upbeat_square_abs3942_2` - i.e. the name of the parent run `upbeat_square_abs3942` followed by **underscore plus a number** (`_2`in this example).
But Azure ML Studio displays names like (no `upbeat_square_abs3942_2` to be found):
So this code works:
```
child_run = mlflow_client.get_run('upbeat_square_abs3942_2')
```
but using a name shown in the screenshot above throws an exception, e.g.
```
child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
```
### Question
How can I obtain the model and metrics for any algorithm used in the experiment?
Thanks!
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 issue 中展示的 MLflow 用戶端用法開始,包括 get_run、automl_best_child_run_id 標籤和 pipeline_id_000。追蹤 AutoML 子執行的識別方式,以及演算法名稱與其內部執行名稱之間的關係。完成的標準是記錄或支援取得實驗中任意演算法的模型及其指標,而不僅僅是最佳執行。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- azure, jupyter-notebook, python
- 領域
- api, machine-learning
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100