Azure / Azure/azure-cli

az ml create update does not return output

Open
#28,476 2 comments 0 reactions 1 assignee Claimed by @banibrata-de View on GitHub
act-codegen-extensibility-squad Auto-Assign bug customer-reported Machine Learning Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

Our pipelines stopped working because the `az ml job create` returns only warnings and no other output. The job is created and running, but we need the output to get the name of the job.
Command is implemented in a Azure DevOps pipeline.

### Related command

```
az ml job create -f train_job.yml \
-g $(azure.resourceGroup) -w $(azureml.WorkspaceName) \
--set trial.environment=azureml:$(amlEnvironmentName)@latest \
--set experiment_name=$(experimentName) \
--set display_name=$(displayName) \
--set compute=azureml:$(azureml.cpuComputeName) --query name -o tsv
```

### Errors

WARNING: Failed to deserialize response: Value 'Null' passed is not in set ['spot', 'basic', 'standard', 'premium']

### Issue script & Debug output

DEBUG: cli.azext_mlv2.manual.custom.utils: Traceback (most recent call last):
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/utils.py", line 62, in _dump_entity_with_warnings
return entity._to_dict() # type: ignore
^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/marshmallow/schema.py", line 517, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/marshmallow/fields.py", line 340, in serialize
return self._serialize(value, attr, obj, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_schema/core/fields.py", line 73, in _serialize
raise ValidationError(f"Value {value!r} passed is not in set {self.allowed_values}")
marshmallow.exceptions.ValidationError: Value 'Null' passed is not in set ['spot', 'basic', 'standard', 'premium']

DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [, ]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute []
INFO: az_command_data_logger: exit code: 0
INFO: cli.__main__: Command ran in 5.133 seconds (init: 0.184, invoke: 4.949)
INFO: cli.azure.cli.core.decorators: Suppress exception:
Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 62, in
raise ex
File "/opt/az/lib/python3.11/site-packages/azure/cli/__main__.py", line 55, in
sys.exit(exit_code)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/decorators.py", line 79, in _wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/telemetry.py", line 532, in _get_secrets_warning_config
show_secrets_warning = _get_config().getboolean('clients', 'show_secrets_warning', fallback=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/knack/config.py", line 147, in getboolean
raise ValueError('Not a boolean: {}'.format(val))
ValueError: Not a boolean: None

INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3470 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /home/vsts/work/_temp/.azclitask"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.

### Expected behavior

No warnings and get the output/response from the command.

### Environment Summary

azure-cli 2.57.0

core 2.57.0
telemetry 1.1.0

Extensions:
azure-devops 1.0.0
ml 2.23.0

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

### Additional context

While using the `--debug` option we can see a response with the information about the job, but most likely because of the warning we cannot get it as normal output of the command. Also `--only-show-errors` does not help. A workaround that works is to use ml version `2.22.0`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.