ISISComputingGroup / ISISComputingGroup/IBEX
Improve monitoring of how long actions are taking
- Lingua principale
- Nessun dato sulla lingua
- Stelle
- 6
- Fork
- 2
- Merge medio
- 16h 40m
- PR unite (30g)
- 2
Descrizione
ENDs on MERLIN have been unpredictably slowing from a script, variable times but often taking up to 10 minutes before the END actually takes effect and is logged by the DAE. Restarting DAE and IOC had no effect on the problem. Before restarting the IBEX server the client was restarted just to check there was nothing going on there.
After a client restart the BEGINS and ENDS behaved normally. The behaviour developed during 6-8hours of the client's use, possibly as a result of a plot or error or some other issue. ICP and DAE IOC logs all looked fine while these problems were occurring. The delays appeared both in the scripted END and when doing a non scripted END on the console.
***Acceptance Criteria***
1. ENDs (and BEGINS etc.) in the client perform a handshake with the server and detect any significant delay in pasing on a control command via a timeout.
2. Genie Python reports on and possibly abandons/errors or retries. the command - depending on what is most helpful.
3. The client is pragmatic and if necessary reports on its poor performance and suggests a restart if it might help.
***Notes***
1) The scientist had had multiple DAE issues over the last few days so it would have been really useful if this had been clearly shown up as a different sort of issue.
2) It might also be useful to also make this separation with other equipment IOCs as problems in client to server communication could affect scripts and timings in a similar way. the DAE was collecting 43uAmps of data instead of the 40 specified in the waitfor command.
3) If there is an issue with the client, this may also need fixing in the script server
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Inizia tracciando come Genie Python invia ENDs e BEGINS degli script e della console attraverso il client e il server degli script a DAE/IOC. Esamina i log di DAE e IOC descritti nell’issue mentre indaghi sui comandi ritardati. Il lavoro è completato quando vengono rilevati e segnalati ritardi significativi nella comunicazione, con un suggerimento appropriato di errore, retry, abbandono o riavvio.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- backend, distributed-systems, observability
- Tipo di issue
- Bug
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 25/100