aws / aws/sagemaker-pytorch-training-toolkit
Worker initialization
- Dominant language
- Python
- Stars
- 202
- Forks
- 96
- PR merge metrics
- No merged PRs in 30d
Description
It seems like there is a bug in initialization logic.
Gunicorn processes are initialized not at container start but at a time of first request arrival.
The global `app` variable [here](https://github.com/aws/sagemaker-pytorch-container/blob/master/src/sagemaker_pytorch_container/serving.py#L101
) is not shared between gunicorn processes, so each process will be initialized only at a request arrival.
This will cause a random behavior. If the request comes to a worker that was already initialized - it will be processed quickly. If the request comes to a worker that is not yet initialized - the response will be delayed for quite some time (>30 sec in my case).
This will even cause `/ping` requests to time out and inability to deploy a container to AWS.
Contributor guide
Research direction
Start by reading the initialization logic around src/sagemaker_pytorch_container/serving.py:101 and trace how workers are started in the container. Reproduce the delayed initialization and /ping timeout behavior if possible; done means workers initialize at container start rather than on their first request, without delayed or timing-out health checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100