ISISComputingGroup / ISISComputingGroup/IBEX

ISISDAE: hang on end

Open
#7,687 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
6
Forks
2
Avg merge
16h 40m
Merged PRs (30d)
2

Description

An END from genie python hung on LET early morning 29/3/2023

Looking at genie python log, end at 07:18:44 timed out in wait for runstate at 07:34
In icp log, no sign of end until 07:35 - begin was at 07:09

dae ioc log - curl error at 07:09 during begin then nothing, “curl_easy_perform() failed: Couldn't connect to server”. Not first time it has done this e.g. 06:53 – previous time this did not stop future commands.

Curl is used to restart the block archiver via the web interface, maybe the restart failure caused the system to hang but needs further investigation

It was reported that runs 91025 and 91026 had funny strange logs, run 91025 started at 00:57 on 29th and 91026 at 06:52

underlying cause is probably the following errors in the block archiver ARBLOCK log
```
[2023-03-29 00:47:22] 2023-03-29 00:47:22.675 INFO [Thread 13459] org.csstudio.archive.engine.scanner.ScanThread (run) - Scan Thread runs
[2023-03-29 00:47:30] Exception in thread "ScanThread" Exception in thread "WriteThread" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
[2023-03-29 00:47:30] java.lang.OutOfMemoryError: Java heap space
[2023-03-29 00:47:52] 2023-03-29 00:47:52.449 SEVERE [Thread 40] com.cosylab.epics.caj.impl.reactor.Reactor (processInternal) -
[2023-03-29 00:47:52] java.lang.OutOfMemoryError: Java heap space
[2023-03-29 00:47:52]
[2023-03-29 00:47:52] 2023-03-29 00:47:52.452:WARN:oejs.HttpChannel:qtp1871612052-12863: ERROR: /group
[2023-03-29 01:06:50] 2023-03-29 01:06:50.948 INFO [Thread 13482] org.csstudio.archive.engine.scanner.ScanThread (run) - Scan Thread runs
[2023-03-29 01:06:53]
[2023-03-29 01:06:53] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "com.cosylab.epics.caj.util.SearchTimer"
[2023-03-29 01:06:57]
[2023-03-29 01:06:57] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Active Thread: Equinox Container: 7a1dbb1b-e786-4588-b57a-a04c5b7c29d3"
[2023-03-29 01:07:02]
[2023-03-29 01:07:02] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1871612052-29"
[2023-03-29 01:07:02]
[2023-03-29 01:07:02] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "WriteThread"
```

Looking at `ArchiveEngine.ini` the process is started with `-Xmx256m` so is limited to 256MB memory.

## Acceptance criteria

* resolve blockserver out of memory issue - change `ArchiveEngine.ini` to pass `-XX:+CrashOnOutOfMemoryError` and hotfix, do same for AlarmServer.ini
* Deploy the corrected ini to all instruments

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.