Lightning-AI / Lightning-AI/pytorch-lightning
Use SIGTERM instead of SIGKILL on DDP Errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
In DDP training, when a process has an exception then a SIGKILL is issued to all other processes to tear them down:
https://github.com/Lightning-AI/lightning/blob/4c3ce605ad814155e309b6bb0737db6b428a2a0c/src/pytorch_lightning/strategies/ddp.py#L458
Because it's a SIGKILL, everything is immediately torn down and no finalizers an run. This is a problem for us, because we have code setup on the main process to report errors (e.g. log the stack trace to our job tracker, upload any dumped files / logs to S3 for permanent storage + post-hoc debugging) which will not run.
### Pitch
The simplest alternative would be to issue a SIGTERM instead of SIGKILL -- this would let us install a signal handler on the main process to do all of the "finalizers" for us before exiting.
### Alternatives
There are other things we could do, of course:
- We could always do something like send SIGTERM, sleep for some time, check that the process is killed, if process is not killed within timeout send SIGKILL. This would guarantee that processes would always be killed in case of a bad SIGTERM handler.
- We could use a `SIGINT` (which raises a `KeyboardInterrupt`) -- that way, all of the `try/except`s and `with` statements will naturally run cleanup without having to write a special handler.
- We could make a Callback that tries to do things in the `on_exception` handler. This should be mostly possible, but would require more work on our side because right now only the main process establishes connections with our services for error handling + logging
### Additional context
_No response_
cc @borda @justusschock @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/pytorch_lightning/strategies/ddp.py at the linked process-teardown code, and trace how DDP errors currently signal the other processes. Resolve whether SIGTERM, SIGINT, or a fallback timeout is the intended behavior; done means error cleanup can run without leaving failed processes behind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100