ISISComputingGroup / ISISComputingGroup/IBEX
ISISDAE: hang on end
- Dominant language
- No language data
- Stars
- 6
- Forks
- 2
- Avg merge
- 16h 40m
- Merged PRs (30d)
- 2
Description
An END from genie python hung on LET early morning 29/3/2023
Looking at genie python log, end at 07:18:44 timed out in wait for runstate at 07:34
In icp log, no sign of end until 07:35 - begin was at 07:09
dae ioc log - curl error at 07:09 during begin then nothing, “curl_easy_perform() failed: Couldn't connect to server”. Not first time it has done this e.g. 06:53 – previous time this did not stop future commands.
Curl is used to restart the block archiver via the web interface, maybe the restart failure caused the system to hang but needs further investigation
It was reported that runs 91025 and 91026 had funny strange logs, run 91025 started at 00:57 on 29th and 91026 at 06:52
underlying cause is probably the following errors in the block archiver ARBLOCK log
```
[2023-03-29 00:47:22] 2023-03-29 00:47:22.675 INFO [Thread 13459] org.csstudio.archive.engine.scanner.ScanThread (run) - Scan Thread runs
[2023-03-29 00:47:30] Exception in thread "ScanThread" Exception in thread "WriteThread" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
[2023-03-29 00:47:30] java.lang.OutOfMemoryError: Java heap space
[2023-03-29 00:47:52] 2023-03-29 00:47:52.449 SEVERE [Thread 40] com.cosylab.epics.caj.impl.reactor.Reactor (processInternal) -
[2023-03-29 00:47:52] java.lang.OutOfMemoryError: Java heap space
[2023-03-29 00:47:52]
[2023-03-29 00:47:52] 2023-03-29 00:47:52.452:WARN:oejs.HttpChannel:qtp1871612052-12863: ERROR: /group
[2023-03-29 01:06:50] 2023-03-29 01:06:50.948 INFO [Thread 13482] org.csstudio.archive.engine.scanner.ScanThread (run) - Scan Thread runs
[2023-03-29 01:06:53]
[2023-03-29 01:06:53] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "com.cosylab.epics.caj.util.SearchTimer"
[2023-03-29 01:06:57]
[2023-03-29 01:06:57] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Active Thread: Equinox Container: 7a1dbb1b-e786-4588-b57a-a04c5b7c29d3"
[2023-03-29 01:07:02]
[2023-03-29 01:07:02] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1871612052-29"
[2023-03-29 01:07:02]
[2023-03-29 01:07:02] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "WriteThread"
```
Looking at `ArchiveEngine.ini` the process is started with `-Xmx256m` so is limited to 256MB memory.
## Acceptance criteria
* resolve blockserver out of memory issue - change `ArchiveEngine.ini` to pass `-XX:+CrashOnOutOfMemoryError` and hotfix, do same for AlarmServer.ini
* Deploy the corrected ini to all instruments
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.