brigadecore / brigadecore/brigade
bug?: artemis primary cannnot see secondary
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 244
- PR merge metrics
- No merged PRs in 30d
Description
Running in an HA configuration, I see this, repeatedly, in the primary's logs:
```
2021-11-13 21:12:00,631 ERROR [org.apache.activemq.artemis.core.client] AMQ214033: Cannot resolve host : java.net.UnknownHostException: brigade-artemis-secondary-0.brigade-artemis-secondary.brigade.svc.cluster.local
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) [rt.jar:1.8.0_312]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) [rt.jar:1.8.0_312]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) [rt.jar:1.8.0_312]
at java.net.InetAddress.getByName(InetAddress.java:1077) [rt.jar:1.8.0_312]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.isSameHostAndPort(NettyConnector.java:1291) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.isEquivalent(NettyConnector.java:1272) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.setBackupConnector(ClientSessionFactoryImpl.java:292) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1484) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.notifyNodeUp(ClientSessionFactoryImpl.java:1497) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.notifyTopologyChange(ActiveMQClientProtocolManager.java:544) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:494) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:820) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:428) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:396) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.19.0.jar:2.19.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.19.0.jar:2.19.0]
```
I've seen failover work before. Not sure yet what's going on here.
Some experimentation may be in order.
Contributor guide
Research direction
Start with the reported HA configuration and the primary logs, then trace the failure through NettyConnector.isSameHostAndPort, ClientSessionFactoryImpl.setBackupConnector, and ServerLocatorImpl.notifyNodeUp. Reproduce the Kubernetes hostname-resolution failure and determine whether the primary can resolve and fail over to the secondary. Done means the underlying cause is identified and primary-to-secondary visibility or failover works reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100