aws / aws/amazon-sagemaker-examples
[Bug Report] Error deploying model on custom container - Worker was terminated due to signal 9
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
[Github Link to Notebook.](https://github.com/migueldft/byo-sac-emails/blob/master/deploy_model.ipynb)
**Describe the bug**
I am working on to deploy my model using a custom container.
I have already managed to fit (train) my model, but cant get to deploy it to an realtime endpoint.
**To reproduce**
Just run the above notebook.
**Logs**
```
2022-05-16T22:20:16.512-03:00 | Starting the inference server with 2 workers.
-- | --
| 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Starting gunicorn 20.1.0
| 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Listening at: unix:/tmp/gunicorn.sock (14)
| 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Using worker: sync
| 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [17] [INFO] Booting worker with pid: 17
| 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [18] [INFO] Booting worker with pid: 18
| 2022-05-16T22:20:58.632-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
| 2022-05-16T22:20:58.632-03:00 | [nltk_data] Unzipping corpora/stopwords.zip.
| 2022-05-16T22:21:01.720-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
| 2022-05-16T22:21:01.720-03:00 | [nltk_data] Package stopwords is already up-to-date!
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:23:34 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:17)
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:17 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:18)
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:18 +0000] [17] [INFO] Worker exiting (pid: 17)
| 2022-05-16T22:24:21.970-03:00 | 2022/05/17 01:24:18 [error] 15#15: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
| 2022-05-16T22:24:21.970-03:00 | 2022/05/17 01:24:18 [error] 15#15: *3 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:18 +0000] [18] [INFO] Worker exiting (pid: 18)
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:20 +0000] [14] [WARNING] Worker with pid 17 was terminated due to signal 9
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [14] [WARNING] Worker with pid 18 was terminated due to signal 9
| 2022-05-16T22:24:21.970-03:00 | 10.32.0.2 - - [17/May/2022:01:24:18 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
| 2022-05-16T22:24:21.970-03:00 | 10.32.0.2 - - [17/May/2022:01:24:18 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [23] [INFO] Booting worker with pid: 23
| 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [24] [INFO] Booting worker with pid: 24
| 2022-05-16T22:24:44.195-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
| 2022-05-16T22:24:44.195-03:00 | [nltk_data] Package stopwords is already up-to-date![nltk_data] Downloading package stopwords to /root/nltk_data...
| 2022-05-16T22:24:44.195-03:00 | [nltk_data] Package stopwords is already up-to-date!
| 2022-05-16T22:25:33.025-03:00 | 10.32.0.2 - - [17/May/2022:01:25:31 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
| 2022-05-16T22:25:33.025-03:00 | 10.32.0.2 - - [17/May/2022:01:25:31 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
| 2022-05-16T22:25:33.025-03:00 | [2022-05-17 01:25:30 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:23)
| 2022-05-16T22:25:33.025-03:00 | [2022-05-17 01:25:31 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:24)
| 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:31 +0000] [24] [INFO] Worker exiting (pid: 24)
| 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:31 +0000] [23] [INFO] Worker exiting (pid: 23)
| 2022-05-16T22:25:33.026-03:00 | 2022/05/17 01:25:31 [error] 15#15: *7 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
| 2022-05-16T22:25:33.026-03:00 | 2022/05/17 01:25:31 [error] 15#15: *5 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
| 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:32 +0000] [14] [WARNING] Worker with pid 23 was terminated due to signal 9
| 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:32 +0000] [14] [WARNING] Worker with pid 24 was terminated due to signal 9
```
Contributor guide
Assessment
This issue has not been assessed yet.