Azure / Azure/MachineLearningNotebooks
azureml.core.Run.log_*() logs are not working in child jobs
- Ngôn ngữ chính
- Jupyter Notebook
- Star
- 4.4k
- Fork
- 2.6k
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Hi everyone,
I am trying to to build an AML pipeline for object detectionc/instance segmentation, where the last component would be used for training and model evaluation.
The pipeline is defined via the YAML format/schema (see below) and is run with `az ml job create --file pipeline.yaml`:
- The pipeline itself is defined as:
```
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
```
- The pipeline components _(Get Data_, _Train_) are defined as:
```
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command
```
I want to highlight/visualize a lot of metrics in the _Metrics_ tab of the component like time-series metrics (loss, f1 etc.), X/Y graphs, confusion matrix etc. As the MLFlow API only support time-series-like metric logging (log a single metric value in each iteration/epoch etc.), for logging more advanced metrics, I try to use the _azureml.core.Run.log*_ interface. The problem is that, these logs are only logged into the _Output + logs_ as json files and not as metrics/graphs into the _Metrics_ tab if they are logged at all. Here are the problematic metric logs:
- `azureml.core.Run.log_table()`: **This is not logged at all, nor into the _Outputs + logs_ tab, nor into the _Metrics_ tab.**
- `azureml.core.Run.log_accuracy_table()`: This is logged only into the _Outputs + logs_ tab as a json file.
```{"schema_type": "accuracy_table", "schema_version": "1.0.1", "data": {"probability_tables": [[[82, 118, 0, 0], [75, 31, 87, 7], [66, 9, 109, 16], [46, 2, 116, 36], [0, 0, 118, 82]], [[60, 140, 0, 0], [56, 20, 120, 4], [47, 4, 136, 13], [28, 0, 140, 32], [0, 0, 140, 60]], [[58, 142, 0, 0], [53, 29, 113, 5], [40, 10, 132, 18], [24, 1, 141, 34], [0, 0, 142, 58]]], "percentile_tables": [[[82, 118, 0, 0], [82, 67, 51, 0], [75, 26, 92, 7], [48, 3, 115, 34], [3, 0, 118, 79]], [[60, 140, 0, 0], [60, 89, 51, 0], [60, 41, 99, 0], [46, 5, 135, 14], [3, 0, 140, 57]], [[58, 142, 0, 0], [56, 93, 49, 2], [54, 47, 95, 4], [41, 10, 132, 17], [3, 0, 142, 55]]], "probability_thresholds": [0.0, 0.25, 0.5, 0.75, 1.0], "percentile_thresholds": [0.0, 0.01, 0.24, 0.98, 1.0], "class_labels": ["class1", "class2", "class3"]}}```
- `azureml.core.Run.log_confusion_matrix()`: This is logged only into the _Outputs + logs_ tab as a json file.
```{"schema_type": "confusion_matrix", "schema_version": "1.0.0", "data": {"class_labels": ["class1", "class2", "class3", "class4"], "matrix": [[4, 0, 1, 9], [0, 0, 0, 1], [6, 0, 5, 0], [0, 0, 0, 1]]}}```
The codes used for these logs are as follows:
```
from azureml.core import Run
...
run = Run.get_context(allow_offline=False)
run.log_table("Y over X", {"x":[1, 2, 3], "y":[0.6, 0.7, 0.89]})
run.log_confusion_matrix(
name="Confusion matrix",
value = {
"schema_type": "confusion_matrix",
"schema_version": "1.0.0",
"data": {
"class_labels": ["class1", "class2", "class3", "class4"],
"matrix": [
[4, 0, 1, 9],
[0, 0, 0, 1],
[6, 0, 5, 0],
[0, 0, 0, 1]
]
}
}
)
run.log_accuracy_table(
name="Accuracy Table",
value= {
"schema_type": "accuracy_table",
"schema_version": "1.0.1",
"data": {
"probability_tables": [
[
[82, 118, 0, 0],
[75, 31, 87, 7],
[66, 9, 109, 16],
[46, 2, 116, 36],
[0, 0, 118, 82]
],
[
[60, 140, 0, 0],
[56, 20, 120, 4],
[47, 4, 136, 13],
[28, 0, 140, 32],
[0, 0, 140, 60]
],
[
[58, 142, 0, 0],
[53, 29, 113, 5],
[40, 10, 132, 18],
[24, 1, 141, 34],
[0, 0, 142, 58]
]
],
"percentile_tables": [
[
[82, 118, 0, 0],
[82, 67, 51, 0],
[75, 26, 92, 7],
[48, 3, 115, 34],
[3, 0, 118, 79]
],
[
[60, 140, 0, 0],
[60, 89, 51, 0],
[60, 41, 99, 0],
[46, 5, 135, 14],
[3, 0, 140, 57]
],
[
[58, 142, 0, 0],
[56, 93, 49, 2],
[54, 47, 95, 4],
[41, 10, 132, 17],
[3, 0, 142, 55]
]
],
"probability_thresholds": [0.0, 0.25, 0.5, 0.75, 1.0],
"percentile_thresholds": [0.0, 0.01, 0.24, 0.98, 1.0],
"class_labels": ["class1", "class2", "class3"]
}
},
description="Some description."
)
```
Here are some screenshots of the Azure ML dashboard.
- The first pic shows that `run.log_accuracy_table()` and `run.log_confusion_matrix()` are logged as json file artifacts but `run.log_table()` is not:

- The second pic shows that neither of the `run.log_*()` metrics are visualized in the _Metrics_ tab:

### IMPORTANT
If I run a simple python script as a job (so no pipeline definitions etc.) the `run.log_accuracy_table()`, `run.log_confusion_matrix()` and `_run.log_table()` metrics are logged properly.

Is this behaviour just a bug related to child jobs?
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.