Deployment isn't working for Status Model.

### Operating System

Windows

### Version Information

Python version 3.8

Name: azure-ai-ml
Version: 0.1.0b6

### Steps to reproduce

we followed Microsoft documentation of Online managed Endpoint
we imported below libraries.

# import required libraries
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Workspace
from azure.ai.ml.entities import (
ManagedOnlineEndpoint,
ManagedOnlineDeployment,
Model,
Environment,
CodeConfiguration,
)

# enter details of your AML workspace
subscription_id =
resource_group =
workspace =

from azure.identity import DefaultAzureCredential
# get a handle to the workspace
ml_client = MLClient(
DefaultAzureCredential(), subscription_id, resource_group, workspace
)

# Creating a unique endpoint name with current datetime to avoid conflicts
import datetime

online_endpoint_name = 'retailendpoint1'

# create an online endpoint
endpoint = ManagedOnlineEndpoint(
name=online_endpoint_name,
description="testing with status model",
auth_mode="key",
tags={"Testing": "Testing with status modell"},
)

ml_client.begin_create_or_update(endpoint)

model = Model(path="./model.pkl") #Model(path="./model.pkl")
env = Environment(
conda_file="./model-1/environment/conda.yaml", #conda_file="./model-1/environment/conda.yml"
image="mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest",
#"mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest"
#"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest"
#mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210727.v1
)

blue_deployment = ManagedOnlineDeployment(
name="blue",
endpoint_name=online_endpoint_name,
model=model,
environment=env,
code_configuration=CodeConfiguration(
code="./model-1/onlinescoring", scoring_script="score.py" #code="./model-1/onlinescoring", scoring_script="score.py"code="./model-1/onlinescoring", scoring_script="score.py"
),
instance_type="Standard_F4s_v2",
instance_count=3,
)

ml_client.begin_create_or_update(blue_deployment)

we used the above code for the deployment. we deployed Sk learn and Tensorflow models so far but we are not able to deploy status models of Time Series Analysis like ARIMA and Exponential Smoothing Holt Winter model. at the moment we need to deploy the Exponential Smoothing Holt Winter Model help to deploy.

### Expected behavior


Check: endpoint retailendpoint1 exists
Uploading onlinescoring (0.01 MBs): 100%|██████████| 5946/5946 [00:00<00:00, 117328.98it/s]

Uploading model.pkl (< 1 MB): 100%|██████████| 2.86M/2.86M [00:00<00:00, 19.6MB/s]

Creating/updating online deployment blue Done (19m 20s)

### Actual behavior

Python 3.8 - AzureML
Cell at position 7 has failed with HttpResponseError: (None) ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready Code: None Message: ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready Exception Details: (None) ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready The build log is available in the workspace blob store "samlopspockgl02" under the path "/azureml/ImageLogs/474fbfba-bd7a-4a10-bc52-c357bee75771/build.log" Code: None Message: ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready The build log is available in the workspace blob store "samlopspockgl02" under the path "/azureml/ImageLogs/474fbfba-bd7a-4a10-bc52-c357bee75771/build.log" File saved
msyed3 · Kernel idle
CPU
0%
RAM
5%
Last saved a few seconds ago
Python 3.8 - AzureML
[1]
12345678910
# import required libraries
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Workspace
from azure.ai.ml.entities import (
ManagedOnlineEndpoint,
ManagedOnlineDeployment,
Model,
Environment,
CodeConfiguration,
)
3 sec
[2]
1234
# enter details of your AML workspace
subscription_id = "eb5a257a-76e2-472d-9fdc-7f9b3172fcf6"
resource_group = "rg-kdevops-dev"
workspace = "amlmlopspockgl02"
<1 sec
[3]
12345
from azure.identity import DefaultAzureCredential
# get a handle to the workspace
ml_client = MLClient(
DefaultAzureCredential(), subscription_id, resource_group, workspace
)
<1 sec
[4]
123456789101112
# Creating a unique endpoint name with current datetime to avoid conflicts
import datetime

online_endpoint_name = 'retailendpoint1'

# create an online endpoint
endpoint = ManagedOnlineEndpoint(
name=online_endpoint_name,
description="testing with Retail model",
auth_mode="key",
tags={"Testing": "Testing with new model, we will terminate this deployement immediately after testing this model"},
)
<1 sec
[5]
1
ml_client.begin_create_or_update(endpoint)
1 min 33 sec
ManagedOnlineEndpoint({'public_network_access': 'Enabled', 'provisioning_state': 'Succeeded', 'scoring_uri': 'https://retailendpoint1.eastus2.inference.ml.azure.com/score', 'swagger_uri': 'https://retailendpoint1.eastus2.inference.ml.azure.com/swagger.json', 'name': 'retailendpoint1', 'description': 'testing with Retail model', 'tags': {'Testing': 'Testing with new model, we will terminate this deployement immediately after testing this model'}, 'properties': {'azureml.onlineendpointid': '/subscriptions/eb5a257a-76e2-472d-9fdc-7f9b3172fcf6/resourcegroups/rg-kdevops-dev/providers/microsoft.machinelearningservices/workspaces/amlmlopspockgl02/onlineendpoints/retailendpoint1', 'AzureAsyncOperationUri': 'https://management.azure.com/subscriptions/eb5a257a-76e2-472d-9fdc-7f9b3172fcf6/providers/Microsoft.MachineLearningServices/locations/eastus2/mfeOperationsStatus/oe:5681d05d-12b4-4b28-88c0-f7713367da4e:e2217163-24f9-4b17-adac-b77bd7ccb858?api-version=2022-02-01-preview'}, 'id': '/subscriptions/eb5a257a-76e2-472d-9fdc-7f9b3172fcf6/resourceGroups/rg-kdevops-dev/providers/Microsoft.MachineLearningServices/workspaces/amlmlopspockgl02/onlineEndpoints/retailendpoint1', 'Resource__source_path': None, 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/msyed3/code/Users/msyed/azureml-examples', 'creation_context': None, 'serialize': , 'auth_mode': 'key', 'location': 'eastus2', 'identity': , 'traffic': {}, 'mirror_traffic': {}, 'kind': 'Managed'})
[6]
1234567891011121314151617181920
model = Model(path="./model.pkl") #Model(path="./model.pkl")
env = Environment(
conda_file="./model-1/environment/conda.yaml", #conda_file="./model-1/environment/conda.yml"
image="mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest",
#"mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest"
#"mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest"
#mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210727.v1
)

blue_deployment = ManagedOnlineDeployment(
name="blue",
endpoint_name=online_endpoint_name,
model=model,
environment=env,
code_configuration=CodeConfiguration(
code="./model-1/onlinescoring", scoring_script="score.py" #code="./model-1/onlinescoring", scoring_script="score.py"code="./model-1/onlinescoring", scoring_script="score.py"
),
instance_type="Standard_F4s_v2",
instance_count=3,
)
<1 sec


[7]
1
ml_client.begin_create_or_update(blue_deployment)

HttpResponseError: (None) ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready Code: None Message: ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready Exception Details: (None) ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready The build log is available in the workspace blob store "samlopspockgl02" under the path "/azureml/ImageLogs/474fbfba-bd7a-4a10-bc52-c357bee75771/build.log" Code: None Message: ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready The build log is available in the workspace blob store "samlopspockgl02" under the path "/azureml/ImageLogs/474fbfba-bd7a-4a10-bc52-c357bee75771/build.log"
Clear cell output
Check: endpoint retailendpoint1 exists
Creating/updating online deployment blue ---------------------------------------------------------------------------
OperationFailed Traceback (most recent call last)
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/base_polling.py:514, in LROBasePolling.run(self)
513 try:
--> 514 self._poll()
516 except BadStatus as err:

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/base_polling.py:554, in LROBasePolling._poll(self)
553 if _failed(self.status()):
--> 554 raise OperationFailed("Operation failed or canceled")
556 final_get_url = self._operation.get_final_get_url(self._pipeline_response)

OperationFailed: Operation failed or canceled

During handling of the above exception, another exception occurred:

HttpResponseError Traceback (most recent call last)
Input In [7], in ()
----> 1 ml_client.begin_create_or_update(blue_deployment)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/_ml_client.py:702, in MLClient.begin_create_or_update(self, entity, **kwargs)
671 def begin_create_or_update(
672 self,
673 entity: Union[
(...)
682 **kwargs,
683 ) -> LROPoller:
684 """Creates or updates an Azure ML resource asynchronously.
685
686 :param entity: The resource to create or update.
(...)
699 azure.ai.ml.entities.BatchEndpoint]]
700 """
--> 702 return _begin_create_or_update(entity, self._operation_container.all_operations, **kwargs)

File /anaconda/envs/azureml_py38/lib/python3.8/functools.py:875, in singledispatch..wrapper(*args, **kw)
871 if not args:
872 raise TypeError(f'{funcname} requires at least '
873 '1 positional argument')
--> 875 return dispatch(args[0].__class__)(*args, **kw)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/_ml_client.py:784, in _(entity, operations, *args, **kwargs)
781 @_begin_create_or_update.register(OnlineDeployment)
782 def _(entity: OnlineDeployment, operations, *args, **kwargs):
783 module_logger.debug("Creating or updating online_deployments")
--> 784 return operations[AzureMLResourceType.ONLINE_DEPLOYMENT].begin_create_or_update(entity, **kwargs)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/_telemetry/activity.py:258, in monitor_with_activity..monitor..wrapper(*args, **kwargs)
255 @functools.wraps(f)
256 def wrapper(*args, **kwargs):
257 with log_activity(logger, activity_name or f.__name__, activity_type, custom_dimensions):
--> 258 return f(*args, **kwargs)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/operations/_online_deployment_operations.py:143, in OnlineDeploymentOperations.begin_create_or_update(self, deployment, local, vscode_debug, no_wait)
140 polling_wait(poller=poller, start_time=start_time, message=message, timeout=None)
142 except Exception as ex:
--> 143 raise ex

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/operations/_online_deployment_operations.py:140, in OnlineDeploymentOperations.begin_create_or_update(self, deployment, local, vscode_debug, no_wait)
138 else:
139 message = f"Creating/updating online deployment {deployment.name} "
--> 140 polling_wait(poller=poller, start_time=start_time, message=message, timeout=None)
142 except Exception as ex:
143 raise ex

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/ai/ml/_utils/_endpoint_utils.py:63, in polling_wait(poller, message, start_time, is_local, timeout)
61 time.sleep(LROConfigurations.SLEEP_TIME)
62 else:
---> 63 poller.result(timeout=timeout)
65 if poller.done():
66 module_logger.warning("Done ")

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/_poller.py:247, in LROPoller.result(self, timeout)
238 def result(self, timeout=None):
239 # type: (Optional[int]) -> PollingReturnType
240 """Return the result of the long running operation, or
241 the result available after the specified timeout.
242
(...)
245 :raises ~azure.core.exceptions.HttpResponseError: Server problem with the query.
246 """
--> 247 self.wait(timeout)
248 return self._polling_method.resource()

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace..decorator..wrapper_use_tracer(*args, **kwargs)
76 span_impl_type = settings.tracing_implementation()
77 if span_impl_type is None:
---> 78 return func(*args, **kwargs)
80 # Merge span is parameter is set, but only if no explicit parent are passed
81 if merge_span and not passed_in_parent:

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/_poller.py:267, in LROPoller.wait(self, timeout)
263 self._thread.join(timeout=timeout)
264 try:
265 # Let's handle possible None in forgiveness here
266 # https://github.com/python/mypy/issues/8165
--> 267 raise self._exception # type: ignore
268 except TypeError: # Was None
269 pass

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/_poller.py:184, in LROPoller._start(self)
177 """Start the long running operation.
178 On completion, runs any callbacks.
179
180 :param callable update_cmd: The API request to check the status of
181 the operation.
182 """
183 try:
--> 184 self._polling_method.run()
185 except AzureError as error:
186 if not error.continuation_token:

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/core/polling/base_polling.py:532, in LROBasePolling.run(self)
525 raise HttpResponseError(
526 response=self._pipeline_response.http_response,
527 message=str(err),
528 error=err
529 )
531 except OperationFailed as err:
--> 532 raise HttpResponseError(
533 response=self._pipeline_response.http_response,
534 error=err
535 )

HttpResponseError: (None) ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready
Code: None
Message: ResourceNotReady: User container has crashed or terminated. Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotready

### Addition information

Conda file

name: model-env
channels:
- defaults
- anaconda
- conda-forge
dependencies:
- python=3.8.5
- statsmodels
- numpy=1.23.4
- pip
- pip:
- azureml-defaults==1.46.0
- scipy==1.5.3
- matplotlib
- patsy==0.5.2
- pandas==1.2.1
- joblib==1.0.0

Scoring Script:

import os
import logging
import json
import numpy
import joblib

def init():
"""
This function is called when the container is initialized/started, typically after create/update of the deployment.
You can write the logic here to perform init operations like caching the model in memory
"""
global model
# AZUREML_MODEL_DIR is an environment variable created during deployment.
# It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)
model_path = os.path.join(
os.getenv("AZUREML_MODEL_DIR"), "model_lstm.pkl"
)
#deserialize the model file back into a sklearn model
model = joblib.load(model_path)
#model = joblib.load(os.path.join('./', "model.pkl"))

logging.info("Init complete")

def run(raw_data):
"""
This function is called for every invocation of the endpoint to perform the actual scoring/prediction.
In the example we extract the data from the json input and call the scikit-learn model's predict()
method and return the result back
"""
logging.info("Request received")
data = json.loads(raw_data)["data"]
data = numpy.array(data)
result = model.forecast(data)
logging.info("Request processed")
return result.tolist()
return True

we attached our model below in a Compressed file for reference.

[Model.zip](https://github.com/Azure/azureml-examples/files/10030988/Model.zip)

Start with the referenced build log at the workspace blob path and inspect model-1/environment/conda.yaml and model-1/onlinescoring/score.py for the container crash. Reproduce the ManagedOnlineDeployment using model.pkl and confirm that the blue deployment completes without ResourceNotReady.

Written by the indexing model from the issue text.