Lightning-AI / Lightning-AI/pytorch-lightning

Use SIGTERM instead of SIGKILL on DDP Errors

Open
#16,285 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature strategy: ddp
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

In DDP training, when a process has an exception then a SIGKILL is issued to all other processes to tear them down:
https://github.com/Lightning-AI/lightning/blob/4c3ce605ad814155e309b6bb0737db6b428a2a0c/src/pytorch_lightning/strategies/ddp.py#L458

Because it's a SIGKILL, everything is immediately torn down and no finalizers an run. This is a problem for us, because we have code setup on the main process to report errors (e.g. log the stack trace to our job tracker, upload any dumped files / logs to S3 for permanent storage + post-hoc debugging) which will not run.

### Pitch

The simplest alternative would be to issue a SIGTERM instead of SIGKILL -- this would let us install a signal handler on the main process to do all of the "finalizers" for us before exiting.

### Alternatives

There are other things we could do, of course:

- We could always do something like send SIGTERM, sleep for some time, check that the process is killed, if process is not killed within timeout send SIGKILL. This would guarantee that processes would always be killed in case of a bad SIGTERM handler.
- We could use a `SIGINT` (which raises a `KeyboardInterrupt`) -- that way, all of the `try/except`s and `with` statements will naturally run cleanup without having to write a special handler.
- We could make a Callback that tries to do things in the `on_exception` handler. This should be mostly possible, but would require more work on our side because right now only the main process establishes connections with our services for error handling + logging

### Additional context

_No response_

cc @borda @justusschock @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/pytorch_lightning/strategies/ddp.py at the linked process-teardown code, and trace how DDP errors currently signal the other processes. Resolve whether SIGTERM, SIGINT, or a fallback timeout is the intended behavior; done means error cleanup can run without leaving failed processes behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.