aws / aws/amazon-sagemaker-examples

[Please Help!] Issue with deploying a custom model pipeline

Open
#1,072 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Hi,
I'm trying to deploy a custom model pipeline using `sagemaker.pipeline.PipelineModel`. The pipeline model includes two parts, raw data preprocessing and inference. I use the build-in sklearn container to do preprocess and a custom lightgbm container to train the model. Below is the sample code:

```
raw_data_preprocess_inferencee_model = sklearn_preprocessor.create_model()
lightgbm_model = clf.create_model()
model_name = 'inference-pipeline-' + timestamp_prefix
endpoint_name = 'inference-pipeline-ep-' + timestamp_prefix
model = PipelineModel(
name=model_name,
role=role,
predictor_cls = sagemaker_session,
models=[
raw_data_preprocess_inferencee_model,
lightgbm_model])
predictor = model.deploy(initial_instance_count=1,
instance_type = 'ml.m4.4xlarge',
endpoint_name=endpoint_name)
```
The lightgbm container is created following this notebook:
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/scikit_bring_your_own

The raw input data preprocessing is created following this one:
https://aws.amazon.com/blogs/machine-learning/preprocess-input-data-before-making-predictions-using-amazon-sagemaker-inference-pipelines-and-scikit-learn/

**Error message:**

> Error hosting endpoint inference-pipeline-ep-2020-03-02-15-52-08: Failed. Reason: The container-2 for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..

**What I did to figure out the problem:**
1. The CloudWatch log of container-2 looks fine:

> 17:00:21
> Starting the inference server with 4 workers.
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Starting gunicorn 20.0.0
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Listening at: unix:/tmp/gunicorn.sock (17)
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Using worker: gevent
> 17:00:21
> [2020-03-02 17:00:18 +0000] [22] [INFO] Booting worker with pid: 22
> 17:00:21
> [2020-03-02 17:00:18 +0000] [23] [INFO] Booting worker with pid: 23
> 17:00:21
> [2020-03-02 17:00:18 +0000] [24] [INFO] Booting worker with pid: 24
> 17:00:21
> [2020-03-02 17:00:18 +0000] [25] [INFO] Booting worker with pid: 25

2. I tried to use an instance with a larger memory (64GB) which gave me the same error.

Please let me know what else you need from me to figure out the problem

-------------------------------------------------------------------------

**Update:**
I can deploy `raw_data_preprocess_inferencee_model` and `lightgbm_model` in two different endpoint without problems

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.