DiamondLightSource / DiamondLightSource/python-dlstbx
Better handle failed processing jobs when terminated by cluster
- Lingua principale
- Python
- Stelle
- 2
- Fork
- 2
- Merge medio
- 1g 11h
- PR unite (30g)
- 2
Descrizione
Raised by Mark; this collection has a permanent spinning cog: https://ispyb.diamond.ac.uk/dc/visit/mx37593-40/id/18159036
This corresponds to a reprocessing run that combined several datasets:

The log file shows it terminating in the middle of integration (I assume due to OOM errors):
```
...
oooooooooooooooooooooooooooooooooo.ooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => good "%" => ok "!" => bad rmsd
"O" => overloaded "#" => many bad "." => weak
"@" => abandoned
Mosaic spread: 0.413 < 0.413 < 0.413
-------------------- Spotfinding SWEEP2 --------------------
67030 spots found on 3600 images (max 1798 / bin)
***
******
********* ****
************* *********
* ******************* *** ************* ***
** ***************************** ******************** ****
************************************************************
************************************************************
************************************************************
************************************************************
1 image 3600
------------------- Autoindexing SWEEP2 --------------------
All possible indexing solutions:
aP 40.40 57.05 77.35 102.77 94.67 100.11
Indexing solution:
aP 40.40 57.05 77.35 102.77 94.67 100.11
-------------------- Integrating SWEEP2 --------------------
```
possibly we need to have a service that occasionally checks the cluster status for known reprocessing jobs, and marks it as failed if the cluster job has gone away but no status update written?
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.