az ml create update does not return output
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
### Describe the bug
Our pipelines stopped working because the `az ml job create` returns only warnings and no other output. The job is created and running, but we need the output to get the name of the job.
Command is implemented in a Azure DevOps pipeline.
### Related command
```
az ml job create -f train_job.yml \
-g $(azure.resourceGroup) -w $(azureml.WorkspaceName) \
--set trial.environment=azureml:$(amlEnvironmentName)@latest \
--set experiment_name=$(experimentName) \
--set display_name=$(displayName) \
--set compute=azureml:$(azureml.cpuComputeName) --query name -o tsv
```
### Errors
WARNING: Failed to deserialize response: Value 'Null' passed is not in set ['spot', 'basic', 'standard', 'premium']
### Issue script & Debug output
DEBUG: cli.azext_mlv2.manual.custom.utils: Traceback (most recent call last):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/utils.py", line 62, in _dump_entity_with_warnings
return entity._to_dict() # type: ignore
^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/marshmallow/schema.py", line 517, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/marshmallow/fields.py", line 340, in serialize
return self._serialize(value, attr, obj, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_schema/core/fields.py", line 73, in _serialize
raise ValidationError(f"Value {value!r} passed is not in set {self.allowed_values}")
marshmallow.exceptions.ValidationError: Value 'Null' passed is not in set ['spot', 'basic', 'standard', 'premium']
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [, ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute []
INFO: az_command_data_logger: exit code: 0
INFO: cli.__main__: Command ran in 5.133 seconds (init: 0.184, invoke: 4.949)
INFO: cli.azure.cli.core.decorators: Suppress exception:
Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 62, in
raise ex
File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 55, in
sys.exit(exit_code)
SystemExit: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/decorators.py", line 79, in _wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/telemetry.py", line 532, in _get_secrets_warning_config
show_secrets_warning = _get_config().getboolean('clients', 'show_secrets_warning', fallback=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/knack/config.py", line 147, in getboolean
raise ValueError('Not a boolean: {}'.format(val))
ValueError: Not a boolean: None
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3470 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /home/vsts/work/_temp/.azclitask"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.
### Expected behavior
No warnings and get the output/response from the command.
### Environment Summary
azure-cli 2.57.0
core 2.57.0
telemetry 1.1.0
Extensions:
azure-devops 1.0.0
ml 2.23.0
Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2
### Additional context
While using the `--debug` option we can see a response with the information about the job, but most likely because of the warning we cannot get it as normal output of the command. Also `--only-show-errors` does not help. A workaround that works is to use ml version `2.22.0`.
Contributor guide
Assessment
This issue has not been assessed yet.