aws / aws/sagemaker-training-toolkit
Pass SIGTERM to training script to stop training
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
**Describe the bug**
SIGTERM from StopTrainingJob doesn't appear to be passed to the training subprocess.
**To reproduce**
Add a SIGTERM handler to a training script, start a training job, then click "Stop". The signal handler will not fire.
**Expected behavior**
Signal handler should fire when "StopTrainingJob" happens
**Screenshots or logs**
If applicable, add screenshots or logs to help explain your problem.
**System information**
A description of your system.
- Include the version of SageMaker Training Toolkit you are using.
- If you are using a [prebuilt Amazon SageMaker Docker image](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html), provide the URL.
- If you are using a custom Docker image, provide:
- framework name (eg. PyTorch)
- framework version
- Python version
- processing unit type (ie. CPU or GPU)
**Additional context**
Add any other context about the problem here.
Contributor guide
Research direction
Reproduce the issue by running a training job with a SIGTERM handler and stopping it through StopTrainingJob. Locate the training subprocess entry point and trace how the stop signal reaches it; the work is done when the handler fires after clicking Stop and the behavior is covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100