Azure / Azure/azure-cli

Not able to invoke batch endpoint with input parameter of type mlflow_model

Open
#27,685 3 comments 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad Auto-Assign bug customer-reported Machine Learning Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

When invoking a batch endpoint with input containing a model reference, the az cli fails, complaining about missing version:
```
$ az ml batch-endpoint invoke --name some-batch-endpoint --file ./payload.yaml --workspace-name my-workspace
Argument '--file' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
$ cat payload.yaml
inputs:
trained_model:
type: mlflow_model
path: azureml://registries/some-registry/models/some-model/versions/42
```
The problem seems to be related only to validation of the input data, and fails even before there is an attempt to invoke the endpoint.

### Related command

az ml batch-endpoint invoke

### Errors

Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

### Issue script & Debug output

```
az ml batch-endpoint invoke --name some-batch-endpoint --file ./payload.yaml --workspace-name my-workspace --debug

cli.knack.cli: Command arguments: ['ml', 'batch-endpoint', 'invoke', '--name', 'some-batch-endpoint', '--file', './payload.yaml', '--workspace-name', 'my-workspace', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [, , ]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
cli.azure.cli.core: ml 7.113 21 146 /Users/ATLEH/.azure/cliextensions/ml
cli.azure.cli.core: Total (1) 7.113 21 146
cli.azure.cli.core: Loaded 21 groups, 146 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : ml batch-endpoint invoke
cli.azure.cli.core: Command table: ml batch-endpoint invoke
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate []
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x105925b40>]
cli.knack.commands: Configured default 'aion-check' for arg resource_group_name
cli.knack.commands: Configured default 'aion-check' for arg workspace_name
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x10594b760>, .add_cache_arguments at 0x10594b880>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [, , .parse_ids_arguments at 0x10594b7f0>]
Argument '--file' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/ATLEH/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /Users/ATLEH/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/3aa4a235-b6e2-48d5-9195-7fcf05b459b0/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
Traceback (most recent call last):
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
return ml_client.batch_endpoints.invoke(
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
return f(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
self._resolve_input(input_data, os.getcwd())
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
raise e
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
cli: None
cli: Traceback (most recent call last):
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
return ml_client.batch_endpoints.invoke(
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
return f(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
self._resolve_input(input_data, os.getcwd())
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
raise e
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
raise ex
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
result = cmd_copy(params)
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
return self.handler(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 303, in ml_batch_endpoint_invoke
log_and_raise_error(err, debug)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error
raise cli_error
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/custom/batch_endpoint.py", line 288, in ml_batch_endpoint_invoke
return ml_client.batch_endpoints.invoke(
File "/usr/local/Cellar/azure-cli/2.53.1/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 275, in wrapper
return f(*args, **kwargs)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 327, in invoke
self._resolve_input(input_data, os.getcwd())
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 517, in _resolve_input
raise e
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_batch_endpoint_operations.py", line 506, in _resolve_input
entry.path = orchestrator.get_asset_arm_id(entry.path, asset_type)
File "/Users/ATLEH/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 187, in get_asset_arm_id
raise ValidationException(
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml.exceptions.ValidationException: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.

cli.azure.cli.core.azclierror: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
az_command_data_logger: Failed to extract version when parsing asset //registries/some-registry/models/some-model/versions/42 of type data as arm id. Version must be provided.
cli.knack.cli: Event: Cli.PostExecute []
```

### Expected behavior

I expect the batch-endpoint to be correctly invoked.

### Environment Summary

```
azure-cli 2.53.1

core 2.53.1
telemetry 1.1.0

Extensions:
ml 2.21.1

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location '/usr/local/Cellar/azure-cli/2.53.1/libexec/bin/python'
Extensions directory '/Users/ATLEH/.azure/cliextensions'

Python (Darwin) 3.10.13 (main, Aug 24 2023, 22:48:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]
```

### Additional context

The same problem appears when using the Azure SDK for Python and calling `ml_client.batch_endpoints.invoke(...)`

Tested with several versions, latest one being [1.11.1](https://github.com/Azure/azure-sdk-for-python).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.