ISISComputingGroup / ISISComputingGroup/IBEX

Improve monitoring of how long actions are taking

Open
#6,469 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
6
Forks
2
Avg merge
16h 40m
Merged PRs (30d)
2

Description

ENDs on MERLIN have been unpredictably slowing from a script, variable times but often taking up to 10 minutes before the END actually takes effect and is logged by the DAE. Restarting DAE and IOC had no effect on the problem. Before restarting the IBEX server the client was restarted just to check there was nothing going on there.

After a client restart the BEGINS and ENDS behaved normally. The behaviour developed during 6-8hours of the client's use, possibly as a result of a plot or error or some other issue. ICP and DAE IOC logs all looked fine while these problems were occurring. The delays appeared both in the scripted END and when doing a non scripted END on the console.

***Acceptance Criteria***
1. ENDs (and BEGINS etc.) in the client perform a handshake with the server and detect any significant delay in pasing on a control command via a timeout.
2. Genie Python reports on and possibly abandons/errors or retries. the command - depending on what is most helpful.
3. The client is pragmatic and if necessary reports on its poor performance and suggests a restart if it might help.

***Notes***
1) The scientist had had multiple DAE issues over the last few days so it would have been really useful if this had been clearly shown up as a different sort of issue.
2) It might also be useful to also make this separation with other equipment IOCs as problems in client to server communication could affect scripts and timings in a similar way. the DAE was collecting 43uAmps of data instead of the 40 specified in the waitfor command.
3) If there is an issue with the client, this may also need fixing in the script server

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.