aws / aws/sagemaker-python-sdk

Multi-model endpoint workers die when sklearn entrypoint imports package installed with requirements.txt

Closed
#1,833 7 comments 0 reactions 1 assignee Claimed by @mollyheamazon View on GitHub
Pending information scikit-learn
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

Multi-model endpoint workers die when the entry point imports a package installed through requirements.txt. The package is installed successfully and the endpoint is created successfully, but inference requests always fail.

```
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "{
"code": 500,
"type": "InternalServerException",
"message": "Worker died."
}
```

**To reproduce**
Include a requirements.txt in the `source_dir` and import the installed package in the entry point script or the `model_fn`.

https://gist.github.com/gavinmh/267bc34ddedaf0931151a901859e165d changes the `sklearn_multi_model_endpoint_home_value.ipynb` example notebook.

In particular, it adds:

```
%%writefile $SOURCE_DIR/requirements.txt

shap
```

**Expected behavior**
`shap` is imported.

**Screenshots or logs**

![image](https://user-images.githubusercontent.com/1227260/90271794-fc83bb00-de21-11ea-8320-b39147a4e422.png)

![image](https://user-images.githubusercontent.com/1227260/90271384-5a63d300-de21-11ea-847d-63df4a66b3a7.png)

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.3.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: sklearn
- **Framework version**: 0.23-1
- **Python version**: 3.7
- **CPU or GPU**: CPU
- **Custom Docker image (Y/N)**: N

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.