Lightning-AI / Lightning-AI/pytorch-lightning

KeyboardInterrupt Signal does not raise KeyboardInterrupt Error

Open
#11,924 11 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deprecation discussion feature pl
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🐛 Bug

If a `KeyboardInterrupt` signal is send during training via e.g. `ctrl-c` the `trainer.fit()` method finishes immediately but no `KeyboardInterrupt` error is raised. Hence, the remaining model training script (everything after the `trainer.fit()` call) continues as if nothing has happened. Since my training script saves a status file "successfull" to disc if no error occured, raising no `KeyboardInterrupt` error during training is quite problematic.

### To Reproduce

The problem comes from the design of the `_call_and_handle_interrupt()` method in the lightning trainer class.
This method wraps the entire training and has the following structure:

```python
import time

def main():
# _call_and_handle_interrupt method which wraps entire training has this structure
# However, the BaseException is never reached if a KeyboardInterrupt is caught and thus no error is raised
try:
# training here
for i in range(100):
time.sleep(1)
except KeyboardInterrupt as e:
print('handle KeyBoardInterrupt.')
# why not raining here as well?
except BaseException as e:
# never reached when a KeyboardInterrupt is caught
print('handle BaseException.')
raise

if __name__ == '__main__':
main()

```

Since the `KeyboardInterrupt` exception handling has no `raise` statement no error is raised and the `trainer.fit()` method exits with no error. It is said that the `KeyboardInterrupt` exception block will be removed in lightning version 1.7 but I think the current behavior is not expected. At least one should raise the caught `KeyboardInterrupt` after handling it.

### Expected behavior

If I send a `KeyboardInterrupt` signal while training, I expect the `trainer.fit()` method to raise that error immediately so that my script finishes without executing further code which relys on the fact that no error occured.

cc @borda @tchaton @rohitgr7 @akihironitta @justusschock @awaelchli @ninginthecloud

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Trainer class's _call_and_handle_interrupt() method and trace how trainer.fit() handles Ctrl-C. Reproduce the interruption during training, then verify that the caught KeyboardInterrupt is propagated so code after trainer.fit() does not run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.