Healthchecks should check db status too
- Dominant language
- Java
- Stars
- 593
- Forks
- 123
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 4
Description
## Gentics Mesh Version, operating system, or hardware.
- v0.31.5
## Operating System
- Linux
## JVM
provided jvm by docker getmesh image
## Problem
Orientdb got closed in the getmesh instance but im unable to find the root cause.
getmesh is still running and any query to an api endpoint that need access to the db will throw this exception:
```
15:15:53.711 [getmesh-master-master-0] ERROR [vert.x-eventloop-thread-2] - Error:
com.orientechnologies.orient.core.exception.ODatabaseException: OrientDB Instance is closed
at com.orientechnologies.orient.core.db.OrientDBEmbedded.checkOpen(OrientDBEmbedded.java:635) ~[mesh.jar:na]
at com.orientechnologies.orient.core.db.OrientDBEmbedded.open(OrientDBEmbedded.java:275) ~[mesh.jar:na]
at com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:205) ~[mesh.jar:na]
at com.orientechnologies.orient.core.db.OrientDB.open(OrientDB.java:190) ~[mesh.jar:na]
at com.gentics.mesh.graphdb.tx.impl.OrientServerStorageImpl.createSession(OrientServerStorageImpl.java:91) ~[mesh.jar:na]
at com.gentics.mesh.graphdb.tx.impl.OrientServerStorageImpl.rawTx(OrientServerStorageImpl.java:52) ~[mesh.jar:na]
at com.syncleus.ferma.ext.orientdb3.OrientDBTx.(OrientDBTx.java:37) ~[mesh.jar:na]
at com.gentics.mesh.graphdb.OrientDBDatabase.tx(OrientDBDatabase.java:957) ~[mesh.jar:na]
at com.gentics.mesh.graphdb.OrientDBDatabase.tx(OrientDBDatabase.java:970) ~[mesh.jar:na]
at com.syncleus.ferma.tx.TxFactory.tx(TxFactory.java:70) ~[mesh.jar:na]
at com.gentics.mesh.router.ProjectsRouter.lambda$addProjectRouter$1(ProjectsRouter.java:85) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.impl.RouterImpl.handleContext(RouterImpl.java:255) [mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.impl.RouterImpl.handleContext(RouterImpl.java:255) [mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:66) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:42) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.doEnd(BodyHandlerImpl.java:231) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.end(BodyHandlerImpl.java:211) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl.lambda$handle$0(BodyHandlerImpl.java:74) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerRequestImpl.handleEnd(HttpServerRequestImpl.java:417) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.handleEnd(Http1xServerConnection.java:482) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.processMessage(Http1xServerConnection.java:456) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:144) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:728) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:619) ~[mesh.jar:na]
at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146) ~[mesh.jar:na]
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337) ~[mesh.jar:na]
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195) ~[mesh.jar:na]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[mesh.jar:na]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) ~[mesh.jar:na]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[mesh.jar:na]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:797) ~[mesh.jar:na]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) ~[mesh.jar:na]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) ~[mesh.jar:na]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[mesh.jar:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
15:15:53.711 [getmesh-master-master-0] ERROR [vert.x-eventloop-thread-0] - Unexpected exception in route
java.lang.IllegalArgumentException: code: -1 (expected: 0+)
at io.netty.handler.codec.http.HttpResponseStatus.(HttpResponseStatus.java:581) ~[mesh.jar:na]
at io.netty.handler.codec.http.HttpResponseStatus.(HttpResponseStatus.java:569) ~[mesh.jar:na]
at io.netty.handler.codec.http.HttpResponseStatus.valueOf(HttpResponseStatus.java:446) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerResponseImpl.setStatusCode(HttpServerResponseImpl.java:112) ~[mesh.jar:na]
at com.gentics.mesh.router.route.FailureHandler.handle(FailureHandler.java:147) ~[mesh.jar:na]
at com.gentics.mesh.router.route.FailureHandler.handle(FailureHandler.java:27) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleFailure(RouteImpl.java:235) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:117) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImpl.doFail(RoutingContextImpl.java:465) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImpl.fail(RoutingContextImpl.java:166) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.fail(RoutingContextWrapper.java:73) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.fail(RoutingContextWrapper.java:73) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.fail(RoutingContextWrapper.java:73) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:128) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.impl.RouterImpl.handleContext(RouterImpl.java:255) [mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.impl.RouterImpl.handleContext(RouterImpl.java:255) [mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:66) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.CookieHandlerImpl.handle(CookieHandlerImpl.java:42) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225) ~[mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120) [mesh.jar:na]
at io.vertx.ext.web.impl.RoutingContextWrapper.next(RoutingContextWrapper.java:149) [mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.doEnd(BodyHandlerImpl.java:231) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.end(BodyHandlerImpl.java:211) ~[mesh.jar:na]
at io.vertx.ext.web.handler.impl.BodyHandlerImpl.lambda$handle$0(BodyHandlerImpl.java:74) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerRequestImpl.handleEnd(HttpServerRequestImpl.java:417) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.handleEnd(Http1xServerConnection.java:482) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.processMessage(Http1xServerConnection.java:456) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:144) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:728) ~[mesh.jar:na]
at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:619) ~[mesh.jar:na]
at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146) ~[mesh.jar:na]
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337) ~[mesh.jar:na]
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195) ~[mesh.jar:na]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[mesh.jar:na]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) ~[mesh.jar:na]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61) ~[mesh.jar:na]
at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[mesh.jar:na]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[mesh.jar:na]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[mesh.jar:na]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[mesh.jar:na]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:797) ~[mesh.jar:na]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) ~[mesh.jar:na]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) ~[mesh.jar:na]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[mesh.jar:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
```
The main problem is that our orchestrator (k8s) check theses endpoints:
- 8081 /api/v1/health/live
- 8081 /api/v1/health/ready
- 8080 /api/v1/
exemple of result:
```
{
meshVersion: "0.31.5",
meshNodeName: "getmesh-staging-master-0",
databaseVendor: "orientdb",
databaseVersion: "3.0.18 - Veloce (build 747595e790a081371496f3bb9c57cec395644d82, branch 3.0.x)",
searchVendor: "elasticsearch",
searchVersion: "6.1.4",
vertxVersion: "3.5.4",
databaseRevision: "46906a4d"
}
```
but theses endpoints still return 200 OK
## Expected behaviour and actual behaviour
/api/v1/health/live should return 5XX error, permitting to k8s to detect a failure and restart getmesh instance.
could be good too to investigate on why getmesh is still running even when embedded db is closed.
Contributor guide
Assessment
This issue has not been assessed yet.