Lightning-AI / Lightning-AI/pytorch-lightning
KeyboardInterrupt Signal does not raise KeyboardInterrupt Error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🐛 Bug
If a `KeyboardInterrupt` signal is send during training via e.g. `ctrl-c` the `trainer.fit()` method finishes immediately but no `KeyboardInterrupt` error is raised. Hence, the remaining model training script (everything after the `trainer.fit()` call) continues as if nothing has happened. Since my training script saves a status file "successfull" to disc if no error occured, raising no `KeyboardInterrupt` error during training is quite problematic.
### To Reproduce
The problem comes from the design of the `_call_and_handle_interrupt()` method in the lightning trainer class.
This method wraps the entire training and has the following structure:
```python
import time
def main():
# _call_and_handle_interrupt method which wraps entire training has this structure
# However, the BaseException is never reached if a KeyboardInterrupt is caught and thus no error is raised
try:
# training here
for i in range(100):
time.sleep(1)
except KeyboardInterrupt as e:
print('handle KeyBoardInterrupt.')
# why not raining here as well?
except BaseException as e:
# never reached when a KeyboardInterrupt is caught
print('handle BaseException.')
raise
if __name__ == '__main__':
main()
```
Since the `KeyboardInterrupt` exception handling has no `raise` statement no error is raised and the `trainer.fit()` method exits with no error. It is said that the `KeyboardInterrupt` exception block will be removed in lightning version 1.7 but I think the current behavior is not expected. At least one should raise the caught `KeyboardInterrupt` after handling it.
### Expected behavior
If I send a `KeyboardInterrupt` signal while training, I expect the `trainer.fit()` method to raise that error immediately so that my script finishes without executing further code which relys on the fact that no error occured.
cc @borda @tchaton @rohitgr7 @akihironitta @justusschock @awaelchli @ninginthecloud
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Trainer class's _call_and_handle_interrupt() method and trace how trainer.fit() handles Ctrl-C. Reproduce the interruption during training, then verify that the caught KeyboardInterrupt is propagated so code after trainer.fit() does not run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100