Saving checkpoints at interrupt

Thank you for providing example implementations!

I was wondering what signal is sent to the docker container when spot training job are interrupted. Is it SIGKILL or SIGTERM with some grace period (https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StopTrainingJob.html)?

I was looking to implement a signal handler which, on SIGTERM, saves the latest checkpoint to S3. That way, resume happens from the exact point in time.
Is this possible? Do we need to account for the time it takes for the uploader service to upload the content of `/opt/ml/checkpoints` to the `checkpoint_s3_uri`?

Any guidelines on how to resume from the latest stop point is much appreciated