ISISComputingGroup / ISISComputingGroup/IBEX

Block Archive: Occasionally restarting

Open
#6,467 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
6
Forks
2
Avg merge
16h 40m
Merged PRs (30d)
2

Description

As an instrument scientist I would like to reliably get the values of my blocks. Currently the block archive engine is dropping out about every 5 minutes with the following error:

```
[2021-04-30 16:49:38] 2021-04-30 16:49:38.342 INFO [Thread 1] org.csstudio.archive.engine.Application (start) - ArchiveEngine ending
[2021-04-30 16:49:38] 2021-04-30 16:49:38.343 INFO [Thread 1] org.csstudio.archive.engine.model.EngineModel (stop) - Stopping scanner
[2021-04-30 16:49:38] 2021-04-30 16:49:38.489 INFO [Thread 207] org.csstudio.archive.engine.scanner.ScanThread (run) - Scan Thread ends
[2021-04-30 16:49:38] 2021-04-30 16:49:38.489 INFO [Thread 1] org.csstudio.archive.engine.model.EngineModel (stop) - Stopping archive groups
[2021-04-30 16:49:38] 2021-04-30 16:49:38.501 INFO [Thread 1] org.csstudio.archive.engine.model.EngineModel (stop) - Stopping writer
[2021-04-30 16:49:38] 2021-04-30 16:49:38.502 INFO [Thread 206] org.csstudio.archive.engine.model.WriteThread (run) - WriteThread exists
[2021-04-30 16:49:38] 2021-04-30 16:49:38.561 INFO [Thread 1] org.csstudio.archive.engine.Application (start) - Reading configuration 'block_engine'
```
This will lead to intermittent data in the database:
![image](https://user-images.githubusercontent.com/10086797/116726203-dff0d880-a9da-11eb-99e9-4f0c83407bd2.png)

Additionally, if this happens at the same time that JSON_bourne is attempting to get data it will then cause the following error:
```
[2021-04-30 16:48:27] 2021-04-30 16:48:27.737 WARNING [Thread 251] org.csstudio.archive.engine.server.Servlet (doGet) - HTTP Server exception
[2021-04-30 16:48:27] java.lang.ArrayIndexOutOfBoundsException: Index 81 out of bounds for length 0

[2021-04-30 16:48:27] at java.base/java.util.concurrent.CopyOnWriteArrayList.elementAt(CopyOnWriteArrayList.java:385)

[2021-04-30 16:48:27] at java.base/java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:398)

[2021-04-30 16:48:27] at org.csstudio.archive.engine.model.ArchiveGroup.getChannel(ArchiveGroup.java:97)

[2021-04-30 16:48:27] at org.csstudio.archive.engine.server.json.JSONGroupResponse.fillResponse(JSONGroupResponse.java:52)

[2021-04-30 16:48:27] at org.csstudio.archive.engine.server.Servlet.doGet(Servlet.java:60)

[2021-04-30 16:48:27] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

[2021-04-30 16:48:27] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl$LegacyServlet.service(HttpServiceRuntimeImpl.java:1240)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.servlet.internal.registration.EndpointRegistration.service(EndpointRegistration.java:151)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.servlet.internal.servlet.ResponseStateHandler.processRequest(ResponseStateHandler.java:65)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.servlet.internal.context.DispatchTargets.doDispatch(DispatchTargets.java:134)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet.service(ProxyServlet.java:103)

[2021-04-30 16:48:27] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

[2021-04-30 16:48:27] at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:300)

[2021-04-30 16:48:27] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)

[2021-04-30 16:48:27] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

[2021-04-30 16:48:27] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.Server.handle(Server.java:502)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)

[2021-04-30 16:48:27] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)

[2021-04-30 16:48:27] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

[2021-04-30 16:48:27] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

[2021-04-30 16:48:27] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)

[2021-04-30 16:48:27] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

[2021-04-30 16:48:27] at java.base/java.lang.Thread.run(Thread.java:834)
```

Which can cause the webdashboard to crash as seen in #6464 .

## Acceptance Criteria
* The block archive engine no longer occasionally fails
* There is a system test that would have covered this

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.