device kernel image is invalid

hi,

i tried to torch.compile the model using: ```model.model_body[0].auto_model = torch.compile(model.model_body[0].auto_model, mode="reduce-overhead")```and get the following error:

error:
```
W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised: | 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
```

used image: [763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu1](https://763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-inference:2.1.0-transformers4.37.0-gpu-py310-cu1)
torch compile: https://pytorch.org/docs/stable/generated/torch.compile.html

full error log:
```
2024-05-06T22:59:03.955-04:00 Collecting scikit-learn==1.2.2 (from -r /opt/ml/model/code/requirements.txt (line 1)) Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting setfit==1.0.3 (from -r /opt/ml/model/code/requirements.txt (line 2)) Downloading setfit-1.0.3-py3-none-any.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting pandas==1.5.3 (from -r /opt/ml/model/code/requirements.txt (line 3)) Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: torch==2.1.0 in /opt/conda/lib/python3.10/site-packages (from -r /opt/ml/model/code/requirements.txt (line 4)) (2.1.0+cu118)
2024-05-06T22:59:03.955-04:00 Collecting optimum (from -r /opt/ml/model/code/requirements.txt (line 5)) Downloading optimum-1.19.1-py3-none-any.whl.metadata (19 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: numpy>=1.17.3 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.22.4)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: scipy>=1.3.2 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.12.0)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: joblib>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (1.3.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn==1.2.2->-r /opt/ml/model/code/requirements.txt (line 1)) (3.2.0)
2024-05-06T22:59:03.955-04:00 Collecting datasets>=2.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading datasets-2.19.1-py3-none-any.whl.metadata (19 kB)
2024-05-06T22:59:03.955-04:00 Collecting sentence-transformers>=2.2.1 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading sentence_transformers-2.7.0-py3-none-any.whl.metadata (11 kB)
2024-05-06T22:59:03.955-04:00 Collecting evaluate>=0.3.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading evaluate-0.4.2-py3-none-any.whl.metadata (9.3 kB)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: huggingface-hub>=0.13.0 in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (0.20.3)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2.8.2)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (2023.4)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.13.1)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (4.9.0)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.12)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.2.1)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (3.1.3)
2024-05-06T22:59:03.955-04:00 Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2023.12.2)
2024-05-06T22:59:03.955-04:00 Collecting coloredlogs (from optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: transformers<4.41.0,>=4.26.0 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (4.37.0)
2024-05-06T22:59:04.348-04:00 Collecting pyarrow>=12.0.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
2024-05-06T22:59:04.348-04:00 Collecting pyarrow-hotfix (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)
2024-05-06T22:59:04.348-04:00 Collecting dill<0.3.9,>=0.3.0 (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2.31.0)
2024-05-06T22:59:04.348-04:00 Requirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (4.64.1)
2024-05-06T22:59:04.599-04:00 Collecting xxhash (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
2024-05-06T22:59:04.849-04:00 Collecting multiprocess (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
2024-05-06T22:59:04.849-04:00 Collecting aiohttp (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)
2024-05-06T22:59:04.849-04:00 Collecting huggingface-hub>=0.13.0 (from setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading huggingface_hub-0.23.0-py3-none-any.whl.metadata (12 kB)
2024-05-06T22:59:05.100-04:00 Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (6.0.1)
2024-05-06T22:59:05.100-04:00 Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas==1.5.3->-r /opt/ml/model/code/requirements.txt (line 3)) (1.16.0)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: Pillow in /opt/conda/lib/python3.10/site-packages (from sentence-transformers>=2.2.1->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (10.2.0)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (2023.12.25)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.15.1)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from transformers<4.41.0,>=4.26.0->transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.4.2)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (0.1.99)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: protobuf in /opt/conda/lib/python3.10/site-packages (from transformers[sentencepiece]<4.41.0,>=4.26.0->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) (3.20.2)
2024-05-06T22:59:05.601-04:00 Collecting humanfriendly>=9.1 (from coloredlogs->optimum->-r /opt/ml/model/code/requirements.txt (line 5)) Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (2.1.4)
2024-05-06T22:59:05.601-04:00 Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.1.0->-r /opt/ml/model/code/requirements.txt (line 4)) (1.3.0)
2024-05-06T22:59:05.601-04:00 Collecting aiosignal>=1.1.2 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
2024-05-06T22:59:05.852-04:00 Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (23.2.0)
2024-05-06T22:59:06.102-04:00 Collecting frozenlist>=1.1.1 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
2024-05-06T22:59:06.102-04:00 Collecting multidict<7.0,>=4.5 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
2024-05-06T22:59:06.352-04:00 Collecting yarl<2.0,>=1.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
2024-05-06T22:59:06.352-04:00 Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.3.2)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (3.4)
2024-05-06T22:59:06.352-04:00 Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (1.26.18)
2024-05-06T22:59:06.607-04:00 Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets>=2.3.0->setfit==1.0.3->-r /opt/ml/model/code/requirements.txt (line 2)) (2023.11.17)
2024-05-06T22:59:06.608-04:00 Downloading scikit_learn-1.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 87.0 MB/s eta 0:00:00
2024-05-06T22:59:06.609-04:00 Downloading setfit-1.0.3-py3-none-any.whl (75 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 38.4 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.1/12.1 MB 130.5 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading optimum-1.19.1-py3-none-any.whl (417 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 417.0/417.0 kB 122.1 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading datasets-2.19.1-py3-none-any.whl (542 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 542.0/542.0 kB 138.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading evaluate-0.4.2-py3-none-any.whl (84 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 43.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading huggingface_hub-0.23.0-py3-none-any.whl (401 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.2/401.2 kB 120.6 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading sentence_transformers-2.7.0-py3-none-any.whl (171 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 171.5/171.5 kB 78.0 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 24.8 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading dill-0.3.8-py3-none-any.whl (116 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 59.0 MB/s eta 0:00:00
2024-05-06T22:59:06.855-04:00 Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 155.8 MB/s eta 0:00:00
2024-05-06T22:59:06.856-04:00 Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 45.1 MB/s eta 0:00:00
2024-05-06T22:59:07.130-04:00 Downloading pyarrow-16.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (40.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 MB 88.9 MB/s eta 0:00:00
2024-05-06T22:59:07.133-04:00 Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 60.0 MB/s eta 0:00:00
2024-05-06T22:59:07.363-04:00 Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
2024-05-06T22:59:07.364-04:00 Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 82.8 MB/s eta 0:00:00
2024-05-06T22:59:07.364-04:00 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
2024-05-06T22:59:07.364-04:00 Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
2024-05-06T22:59:07.364-04:00 Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.5/239.5 kB 71.2 MB/s eta 0:00:00
2024-05-06T22:59:07.364-04:00 Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.3/124.3 kB 49.2 MB/s eta 0:00:00
2024-05-06T22:59:08.868-04:00 Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 301.6/301.6 kB 103.5 MB/s eta 0:00:00
2024-05-06T22:59:12.959-04:00 Installing collected packages: xxhash, pyarrow-hotfix, pyarrow, multidict, humanfriendly, frozenlist, dill, async-timeout, yarl, scikit-learn, pandas, multiprocess, huggingface-hub, coloredlogs, aiosignal, aiohttp, sentence-transformers, datasets, optimum, evaluate, setfit Attempting uninstall: scikit-learn Found existing installation: scikit-learn 1.4.0 Uninstalling scikit-learn-1.4.0: Successfully uninstalled scikit-learn-1.4.0 Attempting uninstall: pandas Found existing installation: pandas 2.2.0 Uninstalling pandas-2.2.0: Successfully uninstalled pandas-2.2.0
2024-05-06T22:59:17.155-04:00 Attempting uninstall: huggingface-hub Found existing installation: huggingface-hub 0.20.3 Uninstalling huggingface-hub-0.20.3: Successfully uninstalled huggingface-hub-0.20.3
2024-05-06T22:59:17.155-04:00 Successfully installed aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 coloredlogs-15.0.1 datasets-2.19.1 dill-0.3.8 evaluate-0.4.2 frozenlist-1.4.1 huggingface-hub-0.23.0 humanfriendly-10.0 multidict-6.0.5 multiprocess-0.70.16 optimum-1.19.1 pandas-1.5.3 pyarrow-16.0.0 pyarrow-hotfix-0.6 scikit-learn-1.2.2 sentence-transformers-2.7.0 setfit-1.0.3 xxhash-3.4.1 yarl-1.9.4
2024-05-06T22:59:17.155-04:00 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
2024-05-06T22:59:17.155-04:00 [notice] A new release of pip is available: 23.3.2 -> 24.0
2024-05-06T22:59:17.406-04:00 [notice] To update, run: pip install --upgrade pip
2024-05-06T22:59:17.656-04:00 Warning: MMS is using non-default JVM parameters: -XX:-UseContainerSupport
2024-05-06T22:59:18.157-04:00 WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-05-06T22:59:18.157-04:00 2024-05-07T02:59:18,138 [INFO ] main com.amazonaws.ml.mms.ModelServer -
2024-05-06T22:59:18.157-04:00 MMS Home: /opt/conda/lib/python3.10/site-packages
2024-05-06T22:59:18.157-04:00 Current directory: /
2024-05-06T22:59:18.157-04:00 Temp directory: /home/model-server/tmp
2024-05-06T22:59:18.157-04:00 Number of GPUs: 1
2024-05-06T22:59:18.157-04:00 Number of CPUs: 4
2024-05-06T22:59:18.157-04:00 Max heap size: 3934 M
2024-05-06T22:59:18.157-04:00 Python executable: /opt/conda/bin/python3.10
2024-05-06T22:59:18.157-04:00 Config file: /etc/sagemaker-mms.properties
2024-05-06T22:59:18.157-04:00 Inference address: http://0.0.0.0:8080
2024-05-06T22:59:18.157-04:00 Management address: http://0.0.0.0:8080
2024-05-06T22:59:18.157-04:00 Model Store: /
2024-05-06T22:59:18.157-04:00 Initial Models: model=/opt/ml/model
2024-05-06T22:59:18.157-04:00 Log dir: null
2024-05-06T22:59:18.157-04:00 Metrics dir: null
2024-05-06T22:59:18.157-04:00 Netty threads: 0
2024-05-06T22:59:18.157-04:00 Netty client threads: 0
2024-05-06T22:59:18.157-04:00 Default workers per model: 1
2024-05-06T22:59:18.157-04:00 Blacklist Regex: N/A
2024-05-06T22:59:18.157-04:00 Maximum Response Size: 6553500
2024-05-06T22:59:18.157-04:00 Maximum Request Size: 6553500
2024-05-06T22:59:18.157-04:00 Preload model: false
2024-05-06T22:59:18.157-04:00 Prefer direct buffer: false
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,145 [INFO ] main com.amazonaws.ml.mms.ModelServer - Loading initial models: /opt/ml/model preload_model: false
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,190 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-model
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,261 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model_service_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker_huggingface_inference_toolkit.handler_service --model-path /opt/ml/model --model-name model --preload-model false --tmp-dir /home/model-server/tmp
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,262 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 67
2024-05-06T22:59:18.408-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MMS worker started.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.10.9
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,263 [INFO ] main com.amazonaws.ml.mms.wlm.ModelManager - Model model loaded.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,267 [INFO ] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,273 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,324 [INFO ] main com.amazonaws.ml.mms.ModelServer - Inference API bind to: http://0.0.0.0:8080
2024-05-06T22:59:18.409-04:00 Model server started.
2024-05-06T22:59:18.409-04:00 2024-05-07T02:59:18,328 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000.
2024-05-06T22:59:21.932-04:00 2024-05-07T02:59:18,329 [WARN ] pool-3-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet.
2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,694 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - PyTorch version 2.1.0+cu118 available.
2024-05-06T22:59:22.179-04:00 2024-05-07T02:59:21,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Load pretrained SentenceTransformer: /opt/ml/model
2024-05-06T22:59:23.697-04:00 2024-05-07T02:59:22,156 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Use pytorch device_name: cuda
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,557 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 18
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,818 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /opt/conda/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
2024-05-06T22:59:23.947-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,819 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - warnings.warn(
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,953 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - device=cuda
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,954 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model loaded!!!
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,955 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Current device - cuda
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,976 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch.compile for model SetFitModel(model_body=SentenceTransformer(
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,977 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
2024-05-06T22:59:24.198-04:00 2024-05-07T02:59:23,978 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - (2): Normalize()
2024-05-06T22:59:24.458-04:00 2024-05-07T02:59:23,980 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ), model_head=LogisticRegression(), multi_target_strategy=None, normalize_embeddings=False, labels=['DOCUMENT', 'POLICY'], model_card_data=SetFitModelCardData(language=None, license=None, tags=['setfit', 'sentence-transformers', 'text-classification', 'generated_from_setfit_trainer'], model_name='SetFit with sentence-transformers/all-MiniLM-L6-v2', model_id=None, dataset_name=None, dataset_id=None, dataset_revision=None, task_name=None, st_id='sentence-transformers/all-MiniLM-L6-v2', hyperparameters={}, eval_results_dict={}, eval_lines_list=[], metric_lines=[], widget=[], predict_example=None, label_example_list=[], tokenizer_warning=False, train_set_metrics_list=[], train_set_sentences_per_label_list=[], code_carbon_callback=None, num_classes=None, best_model_step=None, metrics=['accuracy'], pipeline_tag='text-classification', library_name='setfit', version={'python': '3.10.9', 'setfit': '1.0.3', 'sentence_transformers': '2.7.0', 'transformers': '4.37.0', 'torch': '2.1.0+cu118', 'datasets': '2.19.1', 'tokenizers': '0.15.1'}))
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,347 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model compiled successfully...
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,348 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Model model loaded io_fd=de239efffefa3af1-00000029-00000000-1db2d9548cfdc087-9fd56c90
2024-05-06T22:59:24.460-04:00 2024-05-07T02:59:24,349 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 5989
2024-05-06T22:59:28.210-04:00 2024-05-07T02:59:24,350 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-model-1
2024-05-06T22:59:32.953-04:00 2024-05-07T02:59:28,186 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1
2024-05-06T22:59:38.015-04:00 2024-05-07T02:59:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:42.953-04:00 2024-05-07T02:59:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:47.953-04:00 2024-05-07T02:59:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T22:59:52.953-04:00 2024-05-07T02:59:48,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 1
2024-05-06T22:59:57.953-04:00 2024-05-07T02:59:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:02.992-04:00 2024-05-07T02:59:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:07.952-04:00 2024-05-07T03:00:03,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:12.953-04:00 2024-05-07T03:00:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:42770 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,009 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,012 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json
2024-05-06T23:00:13.092-04:00 2024-05-07T03:00:13,013 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type:
2024-05-06T23:00:13.342-04:00 2024-05-07T03:00:13,020 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:17.953-04:00 2024-05-07T03:00:13,178 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:22.953-04:00 2024-05-07T03:00:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:25.975-04:00 2024-05-07T03:00:23,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,734 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals())
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,770 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles()
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,771 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,772 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,773 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,774 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:00:25.976-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:00:28.232-04:00 2024-05-07T03:00:25,775 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
2024-05-06T23:00:32.953-04:00 2024-05-07T03:00:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:37.953-04:00 2024-05-07T03:00:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:42.953-04:00 2024-05-07T03:00:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:47.953-04:00 2024-05-07T03:00:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:53.000-04:00 2024-05-07T03:00:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:00:57.953-04:00 2024-05-07T03:00:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:02.953-04:00 2024-05-07T03:00:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:07.986-04:00 2024-05-07T03:01:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:12.953-04:00 2024-05-07T03:01:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:17.953-04:00 2024-05-07T03:01:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:22.952-04:00 2024-05-07T03:01:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:27.952-04:00 2024-05-07T03:01:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:32.987-04:00 2024-05-07T03:01:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:37.953-04:00 2024-05-07T03:01:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:42.953-04:00 2024-05-07T03:01:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:47.990-04:00 2024-05-07T03:01:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:52.953-04:00 2024-05-07T03:01:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:01:57.952-04:00 2024-05-07T03:01:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:02.985-04:00 2024-05-07T03:01:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:07.953-04:00 2024-05-07T03:02:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:12.953-04:00 2024-05-07T03:02:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:17.953-04:00 2024-05-07T03:02:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:22.953-04:00 2024-05-07T03:02:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:27.990-04:00 2024-05-07T03:02:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:32.953-04:00 2024-05-07T03:02:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:37.953-04:00 2024-05-07T03:02:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:42.990-04:00 2024-05-07T03:02:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:47.953-04:00 2024-05-07T03:02:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:02:52.953-04:00 2024-05-07T03:02:48,176 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:02:57.952-04:00 2024-05-07T03:02:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:02.953-04:00 2024-05-07T03:02:58,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:07.953-04:00 2024-05-07T03:03:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:12.952-04:00 2024-05-07T03:03:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:17.952-04:00 2024-05-07T03:03:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:22.953-04:00 2024-05-07T03:03:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:27.990-04:00 2024-05-07T03:03:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:32.953-04:00 2024-05-07T03:03:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:37.952-04:00 2024-05-07T03:03:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:42.989-04:00 2024-05-07T03:03:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:47.952-04:00 2024-05-07T03:03:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:52.952-04:00 2024-05-07T03:03:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:03:57.979-04:00 2024-05-07T03:03:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:02.953-04:00 2024-05-07T03:03:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:07.953-04:00 2024-05-07T03:04:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:12.953-04:00 2024-05-07T03:04:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:17.953-04:00 2024-05-07T03:04:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:22.990-04:00 2024-05-07T03:04:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:27.952-04:00 2024-05-07T03:04:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:32.953-04:00 2024-05-07T03:04:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:37.953-04:00 2024-05-07T03:04:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:42.952-04:00 2024-05-07T03:04:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:47.952-04:00 2024-05-07T03:04:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:52.952-04:00 2024-05-07T03:04:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:04:57.988-04:00 2024-05-07T03:04:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:02.952-04:00 2024-05-07T03:04:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:07.953-04:00 2024-05-07T03:05:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:13.052-04:00 2024-05-07T03:05:08,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:17.952-04:00 2024-05-07T03:05:13,174 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:05:22.952-04:00 2024-05-07T03:05:18,173 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:27.985-04:00 2024-05-07T03:05:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:05:32.953-04:00 2024-05-07T03:05:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:37.953-04:00 2024-05-07T03:05:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:42.953-04:00 2024-05-07T03:05:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:47.953-04:00 2024-05-07T03:05:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:52.953-04:00 2024-05-07T03:05:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:05:57.953-04:00 2024-05-07T03:05:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:02.952-04:00 2024-05-07T03:05:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:07.952-04:00 2024-05-07T03:06:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:12.984-04:00 2024-05-07T03:06:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:17.952-04:00 2024-05-07T03:06:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:22.952-04:00 2024-05-07T03:06:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:27.989-04:00 2024-05-07T03:06:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:32.953-04:00 2024-05-07T03:06:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:37.953-04:00 2024-05-07T03:06:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:06:42.953-04:00 2024-05-07T03:06:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:06:47.953-04:00 2024-05-07T03:06:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:52.991-04:00 2024-05-07T03:06:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:06:57.953-04:00 2024-05-07T03:06:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:02.953-04:00 2024-05-07T03:06:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:07.953-04:00 2024-05-07T03:07:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:12.952-04:00 2024-05-07T03:07:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:17.952-04:00 2024-05-07T03:07:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:22.953-04:00 2024-05-07T03:07:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:27.953-04:00 2024-05-07T03:07:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:32.953-04:00 2024-05-07T03:07:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:38.016-04:00 2024-05-07T03:07:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:42.953-04:00 2024-05-07T03:07:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:47.952-04:00 2024-05-07T03:07:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:07:52.994-04:00 2024-05-07T03:07:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:07:57.953-04:00 2024-05-07T03:07:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:02.953-04:00 2024-05-07T03:07:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:07.987-04:00 2024-05-07T03:08:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:12.953-04:00 2024-05-07T03:08:08,175 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:17.953-04:00 2024-05-07T03:08:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:22.953-04:00 2024-05-07T03:08:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:27.953-04:00 2024-05-07T03:08:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:32.987-04:00 2024-05-07T03:08:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:37.953-04:00 2024-05-07T03:08:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:42.953-04:00 2024-05-07T03:08:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:47.952-04:00 2024-05-07T03:08:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:08:52.953-04:00 2024-05-07T03:08:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:08:57.953-04:00 2024-05-07T03:08:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:02.953-04:00 2024-05-07T03:08:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:07.953-04:00 2024-05-07T03:09:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:12.953-04:00 2024-05-07T03:09:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:17.953-04:00 2024-05-07T03:09:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:22.952-04:00 2024-05-07T03:09:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:27.986-04:00 2024-05-07T03:09:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:32.953-04:00 2024-05-07T03:09:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:09:37.953-04:00 2024-05-07T03:09:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:42.988-04:00 2024-05-07T03:09:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:47.952-04:00 2024-05-07T03:09:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:52.952-04:00 2024-05-07T03:09:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:09:57.952-04:00 2024-05-07T03:09:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:02.953-04:00 2024-05-07T03:09:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:07.953-04:00 2024-05-07T03:10:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:10:12.953-04:00 2024-05-07T03:10:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:17.953-04:00 2024-05-07T03:10:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:22.953-04:00 2024-05-07T03:10:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:27.953-04:00 2024-05-07T03:10:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:32.953-04:00 2024-05-07T03:10:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:37.984-04:00 2024-05-07T03:10:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:42.952-04:00 2024-05-07T03:10:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:47.953-04:00 2024-05-07T03:10:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:52.952-04:00 2024-05-07T03:10:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:10:57.953-04:00 2024-05-07T03:10:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:02.993-04:00 2024-05-07T03:10:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:07.952-04:00 2024-05-07T03:11:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:12.953-04:00 2024-05-07T03:11:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:17.953-04:00 2024-05-07T03:11:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:22.953-04:00 2024-05-07T03:11:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:27.953-04:00 2024-05-07T03:11:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:32.953-04:00 2024-05-07T03:11:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:38.019-04:00 2024-05-07T03:11:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:11:42.952-04:00 2024-05-07T03:11:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:47.953-04:00 2024-05-07T03:11:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:11:52.983-04:00 2024-05-07T03:11:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:11:57.953-04:00 2024-05-07T03:11:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:02.952-04:00 2024-05-07T03:11:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:07.987-04:00 2024-05-07T03:12:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:12.953-04:00 2024-05-07T03:12:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:17.952-04:00 2024-05-07T03:12:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:22.952-04:00 2024-05-07T03:12:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:27.953-04:00 2024-05-07T03:12:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:32.988-04:00 2024-05-07T03:12:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:37.953-04:00 2024-05-07T03:12:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:42.953-04:00 2024-05-07T03:12:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:47.953-04:00 2024-05-07T03:12:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:52.953-04:00 2024-05-07T03:12:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:12:57.988-04:00 2024-05-07T03:12:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:02.953-04:00 2024-05-07T03:12:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:07.953-04:00 2024-05-07T03:13:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:12.988-04:00 2024-05-07T03:13:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:17.953-04:00 2024-05-07T03:13:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:22.952-04:00 2024-05-07T03:13:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:27.952-04:00 2024-05-07T03:13:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:32.952-04:00 2024-05-07T03:13:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:13:37.988-04:00 2024-05-07T03:13:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:13:42.952-04:00 2024-05-07T03:13:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:47.953-04:00 2024-05-07T03:13:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:52.991-04:00 2024-05-07T03:13:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:13:57.953-04:00 2024-05-07T03:13:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:02.953-04:00 2024-05-07T03:13:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:07.953-04:00 2024-05-07T03:14:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:12.953-04:00 2024-05-07T03:14:08,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:17.953-04:00 2024-05-07T03:14:13,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:22.953-04:00 2024-05-07T03:14:18,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:27.983-04:00 2024-05-07T03:14:23,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:32.953-04:00 2024-05-07T03:14:28,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:37.953-04:00 2024-05-07T03:14:33,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:42.952-04:00 2024-05-07T03:14:38,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:47.952-04:00 2024-05-07T03:14:43,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:52.989-04:00 2024-05-07T03:14:48,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:14:57.953-04:00 2024-05-07T03:14:53,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:02.952-04:00 2024-05-07T03:14:58,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:03,172 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:58184 "GET /ping HTTP/1.1" 200 1
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - input_data: {"inputs": ["How to create policy?"]}, content_type: application/json
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,343 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - decoded input: {'inputs': ['How to create policy?']}, content_type: application/json
2024-05-06T23:15:06.399-04:00 2024-05-07T03:15:06,344 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - data: {'inputs': ['How to create policy?']}, data_type:
2024-05-06T23:15:08.404-04:00 2024-05-07T03:15:06,346 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:08,177 [INFO ] pool-2-thread-3 ACCESS_LOG - /169.254.178.2:48522 "GET /ping HTTP/1.1" 200 0
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,253 [WARN ] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Batches: 0%| | 0/1 [00:00
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - async_compile.wait(globals())
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1418, in wait
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - scope[key] = result.result()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in result
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel = self.kernel = _load_kernel(self.kernel_name, self.source_code)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1261, in _load_kernel
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - kernel.precompile()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,268 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.launchers = [
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self._precompile_config(c, warm_cache_only_with_cc)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 205, in _precompile_config
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - binary._init_handles()
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/triton/compiler/compiler.py", line 683, in _init_handles
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,269 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/mms/service.py", line 108, in predict
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self._entry_point(input_batch, self.context)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.10/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 267, in handle
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400)
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: backend='inductor' raised:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - RuntimeError: Triton Error [CUDA]: device kernel image is invalid
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - You can suppress this exception and fall back to eager by setting:
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - import torch._dynamo
2024-05-06T23:15:12.412-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - torch._dynamo.config.suppress_errors = True
2024-05-06T23:15:13.420-04:00 2024-05-07T03:15:12,270 [INFO ] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - : 400
```

any idea how this can be fixed? cc @chen3933

Start with the reported model.model_body[0].auto_model torch.compile call and the specified SageMaker Hugging Face inference image. Reproduce the backend='inductor' and Triton CUDA error, then compare the torch, CUDA, GPU image, and installed requirements shown in the full log. Done means identifying a supported configuration or documenting the incompatibility and its scope.

Written by the indexing model from the issue text.