AllenCell / AllenCell/allencell-segmenter-ml
Timeout function in Training causing training to fail
- 主要言語
- Python
- スター
- 5
- フォーク
- 3
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
**Describe the bug**
In Training tab, if the timeout option box is checked, the training will fail. This is an existing bug since Nov 2024 and reproducible on both Mac & Windows OS.
**To Reproduce**
Steps to reproduce the behavior:
1. Start a new model
2. Go through Curation & into Training
3. Fill in all training params as normal, AND check the box to "timeout" and fill in the number of minutes to time out
4. Start training - a popup modal would notify user that training has failed
Console logs:
- [old console log file - Nov 2024](https://docs.google.com/document/d/1k0IquLJHwWwyPbH3PydJb4-3wooYQ7yg_bg1fRGM1Ec/edit?usp=sharing)
- [console log file - Mac](https://docs.google.com/document/d/1ZbHT8NKEzE6zjvURY7di8DfxMW8LWM2lv3Qr1b0v9XI/edit?usp=sharing)
- [console log file - Windows](https://docs.google.com/document/d/1F3iB3b8Zk6NdLwyxoo0jxG97ooP1lkWBmGf0A74j29s/edit?usp=sharing)
**Expected behavior**
Training should run as normal and ends at the timeout specified time.
**Screenshots**
**Describe your data (image format, 2D /3D etc.)** LaminB1 sample dataset
**Environment (please complete the following information):**
- OS: Mac OS 13.6 (22G120), Windows OS built 20348.2655 (EC2 instance)
- Plugin Version: 1.0.0rc8
- PyTorch version 2.0.1 on Windows OS
- GPU? yes on Windows OS
- CUDA version [e.g. 10.0]
**Additional context**
We discussed to remove this feature completely, with the conditions that:
1. User should be able to estimate the time base on how long each epoch might potentially take and set the appropriate number of epoch
2. In case training needs to be stopped before it reach the set number of epoch or before the training is auto-stopped when the model performance is no longer improved, user should be able to cancel the training
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。