ISISComputingGroup / ISISComputingGroup/IBEX

Improve monitoring of how long actions are taking

Aperta
#6,469 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Nessun dato sulla lingua
Stelle
6
Fork
2
Merge medio
16h 40m
PR unite (30g)
2

Descrizione

ENDs on MERLIN have been unpredictably slowing from a script, variable times but often taking up to 10 minutes before the END actually takes effect and is logged by the DAE. Restarting DAE and IOC had no effect on the problem. Before restarting the IBEX server the client was restarted just to check there was nothing going on there.

After a client restart the BEGINS and ENDS behaved normally. The behaviour developed during 6-8hours of the client's use, possibly as a result of a plot or error or some other issue. ICP and DAE IOC logs all looked fine while these problems were occurring. The delays appeared both in the scripted END and when doing a non scripted END on the console.

***Acceptance Criteria***
1. ENDs (and BEGINS etc.) in the client perform a handshake with the server and detect any significant delay in pasing on a control command via a timeout.
2. Genie Python reports on and possibly abandons/errors or retries. the command - depending on what is most helpful.
3. The client is pragmatic and if necessary reports on its poor performance and suggests a restart if it might help.

***Notes***
1) The scientist had had multiple DAE issues over the last few days so it would have been really useful if this had been clearly shown up as a different sort of issue.
2) It might also be useful to also make this separation with other equipment IOCs as problems in client to server communication could affect scripts and timings in a similar way. the DAE was collecting 43uAmps of data instead of the 40 specified in the waitfor command.
3) If there is an issue with the client, this may also need fixing in the script server

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia tracciando come Genie Python invia ENDs e BEGINS degli script e della console attraverso il client e il server degli script a DAE/IOC. Esamina i log di DAE e IOC descritti nell’issue mentre indaghi sui comandi ritardati. Il lavoro è completato quando vengono rilevati e segnalati ritardi significativi nella comunicazione, con un suggerimento appropriato di errore, retry, abbandono o riavvio.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
backend, distributed-systems, observability
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.