Azure / Azure/azureml-examples
batch endpoint samples failing
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
Linux
### Version Information
azure-ai-ml 1.4.0
also failing on azure-ai-ml 1.2
### Steps to reproduce
1) Run any of the batch samples from https://github.com/Azure/azureml-examples/tree/main/sdk/python/endpoints/batch.
Specifically:
text-summarization-batch.ipynb
mlflow-for-batch-tabular.ipynb
custom-output-batch.ipynb
2) All fail on the test stage:
`
job = ml_client.batch_endpoints.invoke(endpoint_name=endpoint.name, input=input)
`
3) Error varies depending on azure-ai-ml package versrion:
e.g. azure-ai-ml 1.2.0 gives:
`File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace..decorator..wrapper_use_tracer(*args, **kwargs)
76 span_impl_type = settings.tracing_implementation()
77 if span_impl_type is None:
---> 78 return func(*args, **kwargs)
80 # Merge span is parameter is set, but only if no explicit parent are passed
81 if merge_span and not passed_in_parent:
File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/azure/ai/ml/operations/_batch_endpoint_operations.py:234, in BatchEndpointOperations.invoke(self, endpoint_name, deployment_name, inputs, **kwargs)
231 self._validate_deployment_name(endpoint_name, deployment_name)
233 if input and isinstance(input, Input):
--> 234 if HTTP_PREFIX not in input.path:
235 self._resolve_input(input, os.getcwd())
236 # MFE expects a dictionary as input_data that's why we are using
237 # "UriFolder" or "UriFile" as keys depending on the input type
TypeError: argument of type 'NoneType' is not iterable`
whereas version 1.4.0 gives:
`
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/azure/ai/ml/_utils/_endpoint_utils.py:116, in validate_response(response)
115 try:
--> 116 r_json = response.json()
117 except ValueError:
118 # exception is not in the json format
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/azure/core/rest/_http_response_impl.py:312, in _HttpResponseBaseImpl.json(self)
311 if not self._json:
--> 312 self._json = loads(self.text())
313 return self._json
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/json/__init__.py:357, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
354 if (cls is None and object_hook is None and
355 parse_int is None and parse_float is None and
356 parse_constant is None and object_pairs_hook is None and not kw):
--> 357 return _default_decoder.decode(s)
358 if cls is None:
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
333 """Return the Python representation of ``s`` (a ``str`` instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
Input In [31], in ()
----> 1 job = ml_client.batch_endpoints.invoke(endpoint_name=endpoint.name, input=input)
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace..decorator..wrapper_use_tracer(*args, **kwargs)
76 span_impl_type = settings.tracing_implementation()
77 if span_impl_type is None:
---> 78 return func(*args, **kwargs)
80 # Merge span is parameter is set, but only if no explicit parent are passed
81 if merge_span and not passed_in_parent:
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/azure/ai/ml/operations/_batch_endpoint_operations.py:318, in BatchEndpointOperations.invoke(self, endpoint_name, deployment_name, inputs, **kwargs)
311 headers[EndpointInvokeFields.MODEL_DEPLOYMENT] = deployment_name
313 response = self._requests_pipeline.post(
314 endpoint.properties.scoring_uri,
315 json=BatchJobResource(properties=batch_job).serialize(),
316 headers=headers,
317 )
--> 318 validate_response(response)
319 batch_job = json.loads(response.text())
320 return BatchJobResource.deserialize(batch_job)
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/azure/ai/ml/_utils/_endpoint_utils.py:119, in validate_response(response)
116 r_json = response.json()
117 except ValueError:
118 # exception is not in the json format
--> 119 raise Exception(response.content.decode("utf-8"))
120 failure_msg = r_json.get("error", {}).get("message", response)
121 error_map = {
122 401: ClientAuthenticationError,
123 404: ResourceNotFoundError,
124 409: ResourceExistsError,
125 }
Exception: BY_POLICY
`
Have tried different Python versions, Conda envs, public workspace vs private workspace.
### Expected behavior
API call should return job object to be used downstream.
### Actual behavior
See error above
### Addition information
_No response_
Contributor guide
Research direction
Start with text-summarization-batch.ipynb, mlflow-for-batch-tabular.ipynb, and custom-output-batch.ipynb, then run the batch endpoint invoke call described in the issue. Compare the reported SDK-version errors and the BY_POLICY response; the work is done when the samples return a job object for downstream use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- api, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100