Azure / Azure/MachineLearningNotebooks

azureml.core.Run.log_*() logs are not working in child jobs

Abierto
#1,922 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Jupyter Notebook
Estrellas
4.4k
Forks
2.6k
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi everyone,

I am trying to to build an AML pipeline for object detectionc/instance segmentation, where the last component would be used for training and model evaluation.

The pipeline is defined via the YAML format/schema (see below) and is run with `az ml job create --file pipeline.yaml`:

- The pipeline itself is defined as:
```
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
```
- The pipeline components _(Get Data_, _Train_) are defined as:
```
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command
```

I want to highlight/visualize a lot of metrics in the _Metrics_ tab of the component like time-series metrics (loss, f1 etc.), X/Y graphs, confusion matrix etc. As the MLFlow API only support time-series-like metric logging (log a single metric value in each iteration/epoch etc.), for logging more advanced metrics, I try to use the _azureml.core.Run.log*_ interface. The problem is that, these logs are only logged into the _Output + logs_ as json files and not as metrics/graphs into the _Metrics_ tab if they are logged at all. Here are the problematic metric logs:

- `azureml.core.Run.log_table()`: **This is not logged at all, nor into the _Outputs + logs_ tab, nor into the _Metrics_ tab.**
- `azureml.core.Run.log_accuracy_table()`: This is logged only into the _Outputs + logs_ tab as a json file.
```{"schema_type": "accuracy_table", "schema_version": "1.0.1", "data": {"probability_tables": [[[82, 118, 0, 0], [75, 31, 87, 7], [66, 9, 109, 16], [46, 2, 116, 36], [0, 0, 118, 82]], [[60, 140, 0, 0], [56, 20, 120, 4], [47, 4, 136, 13], [28, 0, 140, 32], [0, 0, 140, 60]], [[58, 142, 0, 0], [53, 29, 113, 5], [40, 10, 132, 18], [24, 1, 141, 34], [0, 0, 142, 58]]], "percentile_tables": [[[82, 118, 0, 0], [82, 67, 51, 0], [75, 26, 92, 7], [48, 3, 115, 34], [3, 0, 118, 79]], [[60, 140, 0, 0], [60, 89, 51, 0], [60, 41, 99, 0], [46, 5, 135, 14], [3, 0, 140, 57]], [[58, 142, 0, 0], [56, 93, 49, 2], [54, 47, 95, 4], [41, 10, 132, 17], [3, 0, 142, 55]]], "probability_thresholds": [0.0, 0.25, 0.5, 0.75, 1.0], "percentile_thresholds": [0.0, 0.01, 0.24, 0.98, 1.0], "class_labels": ["class1", "class2", "class3"]}}```
- `azureml.core.Run.log_confusion_matrix()`: This is logged only into the _Outputs + logs_ tab as a json file.
```{"schema_type": "confusion_matrix", "schema_version": "1.0.0", "data": {"class_labels": ["class1", "class2", "class3", "class4"], "matrix": [[4, 0, 1, 9], [0, 0, 0, 1], [6, 0, 5, 0], [0, 0, 0, 1]]}}```

The codes used for these logs are as follows:

```
from azureml.core import Run
...

run = Run.get_context(allow_offline=False)
run.log_table("Y over X", {"x":[1, 2, 3], "y":[0.6, 0.7, 0.89]})
run.log_confusion_matrix(
name="Confusion matrix",
value = {
"schema_type": "confusion_matrix",
"schema_version": "1.0.0",
"data": {
"class_labels": ["class1", "class2", "class3", "class4"],
"matrix": [
[4, 0, 1, 9],
[0, 0, 0, 1],
[6, 0, 5, 0],
[0, 0, 0, 1]
]
}
}
)
run.log_accuracy_table(
name="Accuracy Table",
value= {
"schema_type": "accuracy_table",
"schema_version": "1.0.1",
"data": {
"probability_tables": [
[
[82, 118, 0, 0],
[75, 31, 87, 7],
[66, 9, 109, 16],
[46, 2, 116, 36],
[0, 0, 118, 82]
],
[
[60, 140, 0, 0],
[56, 20, 120, 4],
[47, 4, 136, 13],
[28, 0, 140, 32],
[0, 0, 140, 60]
],
[
[58, 142, 0, 0],
[53, 29, 113, 5],
[40, 10, 132, 18],
[24, 1, 141, 34],
[0, 0, 142, 58]
]
],
"percentile_tables": [
[
[82, 118, 0, 0],
[82, 67, 51, 0],
[75, 26, 92, 7],
[48, 3, 115, 34],
[3, 0, 118, 79]
],
[
[60, 140, 0, 0],
[60, 89, 51, 0],
[60, 41, 99, 0],
[46, 5, 135, 14],
[3, 0, 140, 57]
],
[
[58, 142, 0, 0],
[56, 93, 49, 2],
[54, 47, 95, 4],
[41, 10, 132, 17],
[3, 0, 142, 55]
]
],
"probability_thresholds": [0.0, 0.25, 0.5, 0.75, 1.0],
"percentile_thresholds": [0.0, 0.01, 0.24, 0.98, 1.0],
"class_labels": ["class1", "class2", "class3"]
}
},
description="Some description."
)
```

Here are some screenshots of the Azure ML dashboard.

- The first pic shows that `run.log_accuracy_table()` and `run.log_confusion_matrix()` are logged as json file artifacts but `run.log_table()` is not:
![AML_outputs_tab](https://github.com/Azure/MachineLearningNotebooks/assets/30471303/c90fc8e9-f61b-4076-b84b-6a25165157cc)
- The second pic shows that neither of the `run.log_*()` metrics are visualized in the _Metrics_ tab:
![AML_metrics_tab](https://github.com/Azure/MachineLearningNotebooks/assets/30471303/6fc15758-cb40-465c-ac9b-63756f31c7c2)

### IMPORTANT
If I run a simple python script as a job (so no pipeline definitions etc.) the `run.log_accuracy_table()`, `run.log_confusion_matrix()` and `_run.log_table()` metrics are logged properly.

![aml-simple-job](https://github.com/Azure/MachineLearningNotebooks/assets/30471303/ab27f0b4-d88e-4631-abf5-0a1b22ec4ede)

Is this behaviour just a bug related to child jobs?

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.