aws / aws/sagemaker-pytorch-inference-toolkit

MultiDataModel error during prediction: Please provide a model_fn implementation.

Open
#92 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When deploying a packaged PyTorch model using the `PyTorchModel` class I can successfully deploy and call the predict function, but as soon as I use the same model and pass it to a `MultiDataModel` class, the deployment process goes through, but when I call `predictor.predict(data=data, target_model='model.tar.gz')` I get the following error:

> An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from model with message "Your invocation timed out while waiting for a response from container model. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.".

I'm not sure if the error is related to the 'Please provide a model_fn implementation.' error I get in cloudwatch, but the `model_fn` function is in actually implemented and `MultiDataModel` somehow doesn't load it.

**To reproduce**
1. create a sample PyTorch model, train and package it.
2. deploy the model using `PyTorchModel`: (This will successfully deploy the model and when calling `predictor.predict()` successfully returns the inference results.

```
class InvoiceExtraction(RealTimePredictor):
def __init__(self, endpoint_name, sagemaker_session):
super().__init__(endpoint_name, sagemaker_session=sagemaker_session, serializer=json_serializer,
deserializer=json_deserializer, content_type='application/json')

model = PyTorchModel(model_data=str('/home/ec2-user/SageMaker/model.tar.gz'),
name=name_from_base(MODEL_NAME),
role=role,
entry_point='predictor.py',
framework_version='1.5.0', # Breaks for 1.6.0
py_version='py3',
predictor_cls=InvoiceExtraction)

predictor = model.deploy(initial_instance_count=1, instance_type='ml.m5.xlarge', endpoint_name=ENDPOINT_NAME)
predicted_value = predictor.predict(data=data)
```

3. if you deploy the model using a MultiDataModel instead, it will get deployed, but the predict function returns the error mentioned above.

```
model_data_prefix = 's3://multi-model-endpoint-models/'
model.sagemaker_session = sagemaker_session # not setting this results in the model's session not being initialized
mme = MultiDataModel(name=MODEL_NAME,
model_data_prefix=model_data_prefix,
model=model,# passing our pytorch model
sagemaker_session=sagemaker_session)

ENDPOINT_INSTANCE_TYPE = 'ml.m4.xlarge'
ENDPOINT_NAME = 'test-endpoint'

predictor = mme.deploy(initial_instance_count=1,
instance_type=ENDPOINT_INSTANCE_TYPE,
endpoint_name=ENDPOINT_NAME)

mme.add_model(model_data_source='/home/ec2-user/SageMaker/model.tar.gz', model_data_path='model.tar.gz')
list(mme.list_models())

predicted_value = predictor.predict(data=data, target_model='model.tar.gz')
```

**Expected behavior**
`MultiDataModel` should deploy and work without any errors.

**Screenshots or logs**
This is what's included in the CloudWatch logs:

> 2021-02-02 20:24:54,652 [INFO ] W-9000-2093075ac497ff81bd6238817 com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 1
>
> 2021-02-02 20:24:54,653 [WARN ] W-9000-2093075ac497ff81bd6238817 com.amazonaws.ml.mms.wlm.WorkerThread - Backend worker thread exception.
>
> java.lang.IllegalArgumentException: reasonPhrase contains one of the following prohibited characters: \r\n:
>
> Please provide a model_fn implementation.
>
> See documentation for model_fn at https://github.com/aws/sagemaker-python-sdk
>
> #011at io.netty.handler.codec.http.HttpResponseStatus.(HttpResponseStatus.java:555)
>
> #011at io.netty.handler.codec.http.HttpResponseStatus.(HttpResponseStatus.java:537)
>
> #011at io.netty.handler.codec.http.HttpResponseStatus.valueOf(HttpResponseStatus.java:465)
>
> #011at com.amazonaws.ml.mms.wlm.Job.response(Job.java:85)
>
> #011at com.amazonaws.ml.mms.wlm.BatchAggregator.sendResponse(BatchAggregator.java:85)
>
> #011at com.amazonaws.ml.mms.wlm.WorkerThread.run(WorkerThread.java:146)
>
> #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> #011at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> #011at java.lang.Thread.run(Thread.java:748)
>
> 2021-02-02 20:24:54,655 [ERROR] W-9000-2093075ac497ff81bd6238817 com.amazonaws.ml.mms.wlm.BatchAggregator - Unexpected job: 99ef86e2-aedf-47d7-8f6c-950fde1bec88

**System information**
A description of your system. Please provide:
- **Toolkit version**: Latest
- **Framework version**: tried both on 1.5.0 and 1.6.0
- **Python version**: 3.6
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start with the predictor.py entry point and compare the working PyTorchModel deployment with the MultiDataModel path that passes model.tar.gz as target_model. Review how MultiDataModel loads the packaged entry point and how the "Please provide a model_fn implementation" message is produced in the inference container logs. Done means the reproduced MultiDataModel deployment returns inference results without the model_fn error or timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.