aws / aws/amazon-sagemaker-examples
[Please Help!] Issue with deploying a custom model pipeline
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Hi,
I'm trying to deploy a custom model pipeline using `sagemaker.pipeline.PipelineModel`. The pipeline model includes two parts, raw data preprocessing and inference. I use the build-in sklearn container to do preprocess and a custom lightgbm container to train the model. Below is the sample code:
```
raw_data_preprocess_inferencee_model = sklearn_preprocessor.create_model()
lightgbm_model = clf.create_model()
model_name = 'inference-pipeline-' + timestamp_prefix
endpoint_name = 'inference-pipeline-ep-' + timestamp_prefix
model = PipelineModel(
name=model_name,
role=role,
predictor_cls = sagemaker_session,
models=[
raw_data_preprocess_inferencee_model,
lightgbm_model])
predictor = model.deploy(initial_instance_count=1,
instance_type = 'ml.m4.4xlarge',
endpoint_name=endpoint_name)
```
The lightgbm container is created following this notebook:
https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/scikit_bring_your_own
The raw input data preprocessing is created following this one:
https://aws.amazon.com/blogs/machine-learning/preprocess-input-data-before-making-predictions-using-amazon-sagemaker-inference-pipelines-and-scikit-learn/
**Error message:**
> Error hosting endpoint inference-pipeline-ep-2020-03-02-15-52-08: Failed. Reason: The container-2 for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..
**What I did to figure out the problem:**
1. The CloudWatch log of container-2 looks fine:
> 17:00:21
> Starting the inference server with 4 workers.
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Starting gunicorn 20.0.0
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Listening at: unix:/tmp/gunicorn.sock (17)
> 17:00:21
> [2020-03-02 17:00:18 +0000] [17] [INFO] Using worker: gevent
> 17:00:21
> [2020-03-02 17:00:18 +0000] [22] [INFO] Booting worker with pid: 22
> 17:00:21
> [2020-03-02 17:00:18 +0000] [23] [INFO] Booting worker with pid: 23
> 17:00:21
> [2020-03-02 17:00:18 +0000] [24] [INFO] Booting worker with pid: 24
> 17:00:21
> [2020-03-02 17:00:18 +0000] [25] [INFO] Booting worker with pid: 25
2. I tried to use an instance with a larger memory (64GB) which gave me the same error.
Please let me know what else you need from me to figure out the problem
-------------------------------------------------------------------------
**Update:**
I can deploy `raw_data_preprocess_inferencee_model` and `lightgbm_model` in two different endpoint without problems
Contributor guide
Assessment
This issue has not been assessed yet.