DiamondLightSource / DiamondLightSource/python-dlstbx

Better handle failed processing jobs when terminated by cluster

Đang mở
#289 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
2
Fork
2
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
2

Mô tả

Raised by Mark; this collection has a permanent spinning cog: https://ispyb.diamond.ac.uk/dc/visit/mx37593-40/id/18159036

This corresponds to a reprocessing run that combined several datasets:

![Image](https://github.com/user-attachments/assets/bbad1ac7-bdf0-40d0-820a-58afa91c9597)

The log file shows it terminating in the middle of integration (I assume due to OOM errors):
```
...
oooooooooooooooooooooooooooooooooo.ooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
"o" => good "%" => ok "!" => bad rmsd
"O" => overloaded "#" => many bad "." => weak
"@" => abandoned
Mosaic spread: 0.413 < 0.413 < 0.413
-------------------- Spotfinding SWEEP2 --------------------
67030 spots found on 3600 images (max 1798 / bin)
***
******
********* ****
************* *********
* ******************* *** ************* ***
** ***************************** ******************** ****
************************************************************
************************************************************
************************************************************
************************************************************
1 image 3600
------------------- Autoindexing SWEEP2 --------------------
All possible indexing solutions:
aP 40.40 57.05 77.35 102.77 94.67 100.11
Indexing solution:
aP 40.40 57.05 77.35 102.77 94.67 100.11
-------------------- Integrating SWEEP2 --------------------
```

possibly we need to have a service that occasionally checks the cluster status for known reprocessing jobs, and marks it as failed if the cluster job has gone away but no status update written?

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.