AllenCell / AllenCell/allencell-segmenter-ml

Timeout function in Training causing training to fail

Offen
#627 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
5
Forks
3
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

**Describe the bug**
In Training tab, if the timeout option box is checked, the training will fail. This is an existing bug since Nov 2024 and reproducible on both Mac & Windows OS.

**To Reproduce**
Steps to reproduce the behavior:
1. Start a new model
2. Go through Curation & into Training
3. Fill in all training params as normal, AND check the box to "timeout" and fill in the number of minutes to time out
4. Start training - a popup modal would notify user that training has failed

Console logs:
- [old console log file - Nov 2024](https://docs.google.com/document/d/1k0IquLJHwWwyPbH3PydJb4-3wooYQ7yg_bg1fRGM1Ec/edit?usp=sharing)
- [console log file - Mac](https://docs.google.com/document/d/1ZbHT8NKEzE6zjvURY7di8DfxMW8LWM2lv3Qr1b0v9XI/edit?usp=sharing)
- [console log file - Windows](https://docs.google.com/document/d/1F3iB3b8Zk6NdLwyxoo0jxG97ooP1lkWBmGf0A74j29s/edit?usp=sharing)

**Expected behavior**
Training should run as normal and ends at the timeout specified time.

**Screenshots**
Image

**Describe your data (image format, 2D /3D etc.)** LaminB1 sample dataset

**Environment (please complete the following information):**
- OS: Mac OS 13.6 (22G120), Windows OS built 20348.2655 (EC2 instance)
- Plugin Version: 1.0.0rc8
- PyTorch version 2.0.1 on Windows OS
- GPU? yes on Windows OS
- CUDA version [e.g. 10.0]

**Additional context**
We discussed to remove this feature completely, with the conditions that:
1. User should be able to estimate the time base on how long each epoch might potentially take and set the appropriate number of epoch
2. In case training needs to be stopped before it reach the set number of epoch or before the training is auto-stopped when the model performance is no longer improved, user should be able to cancel the training

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.