aws / aws/amazon-sagemaker-examples

[Bug Report] Error deploying model on custom container - Worker was terminated due to signal 9

Open
#3,407 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[Github Link to Notebook.](https://github.com/migueldft/byo-sac-emails/blob/master/deploy_model.ipynb)

**Describe the bug**
I am working on to deploy my model using a custom container.
I have already managed to fit (train) my model, but cant get to deploy it to an realtime endpoint.

**To reproduce**
Just run the above notebook.

**Logs**

```

2022-05-16T22:20:16.512-03:00 | Starting the inference server with 2 workers.
-- | --
  | 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Starting gunicorn 20.1.0
  | 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Listening at: unix:/tmp/gunicorn.sock (14)
  | 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [14] [INFO] Using worker: sync
  | 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [17] [INFO] Booting worker with pid: 17
  | 2022-05-16T22:20:16.513-03:00 | [2022-05-17 01:20:16 +0000] [18] [INFO] Booting worker with pid: 18
  | 2022-05-16T22:20:58.632-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
  | 2022-05-16T22:20:58.632-03:00 | [nltk_data] Unzipping corpora/stopwords.zip.
  | 2022-05-16T22:21:01.720-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
  | 2022-05-16T22:21:01.720-03:00 | [nltk_data] Package stopwords is already up-to-date!
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:23:34 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:17)
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:17 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:18)
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:18 +0000] [17] [INFO] Worker exiting (pid: 17)
  | 2022-05-16T22:24:21.970-03:00 | 2022/05/17 01:24:18 [error] 15#15: *1 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
  | 2022-05-16T22:24:21.970-03:00 | 2022/05/17 01:24:18 [error] 15#15: *3 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:18 +0000] [18] [INFO] Worker exiting (pid: 18)
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:20 +0000] [14] [WARNING] Worker with pid 17 was terminated due to signal 9
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [14] [WARNING] Worker with pid 18 was terminated due to signal 9
  | 2022-05-16T22:24:21.970-03:00 | 10.32.0.2 - - [17/May/2022:01:24:18 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
  | 2022-05-16T22:24:21.970-03:00 | 10.32.0.2 - - [17/May/2022:01:24:18 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [23] [INFO] Booting worker with pid: 23
  | 2022-05-16T22:24:21.970-03:00 | [2022-05-17 01:24:21 +0000] [24] [INFO] Booting worker with pid: 24
  | 2022-05-16T22:24:44.195-03:00 | [nltk_data] Downloading package stopwords to /root/nltk_data...
  | 2022-05-16T22:24:44.195-03:00 | [nltk_data] Package stopwords is already up-to-date![nltk_data] Downloading package stopwords to /root/nltk_data...
  | 2022-05-16T22:24:44.195-03:00 | [nltk_data] Package stopwords is already up-to-date!
  | 2022-05-16T22:25:33.025-03:00 | 10.32.0.2 - - [17/May/2022:01:25:31 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
  | 2022-05-16T22:25:33.025-03:00 | 10.32.0.2 - - [17/May/2022:01:25:31 +0000] "GET /ping HTTP/1.1" 502 173 "-" "AHC/2.0"
  | 2022-05-16T22:25:33.025-03:00 | [2022-05-17 01:25:30 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:23)
  | 2022-05-16T22:25:33.025-03:00 | [2022-05-17 01:25:31 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:24)
  | 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:31 +0000] [24] [INFO] Worker exiting (pid: 24)
  | 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:31 +0000] [23] [INFO] Worker exiting (pid: 23)
  | 2022-05-16T22:25:33.026-03:00 | 2022/05/17 01:25:31 [error] 15#15: *7 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
  | 2022-05-16T22:25:33.026-03:00 | 2022/05/17 01:25:31 [error] 15#15: *5 upstream prematurely closed connection while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
  | 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:32 +0000] [14] [WARNING] Worker with pid 23 was terminated due to signal 9
  | 2022-05-16T22:25:33.026-03:00 | [2022-05-17 01:25:32 +0000] [14] [WARNING] Worker with pid 24 was terminated due to signal 9

 
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.