brigadecore / brigadecore/brigade

bug?: artemis primary cannnot see secondary

Open
#1,740 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
2.4k
Forks
244
PR merge metrics
No merged PRs in 30d

Description

Running in an HA configuration, I see this, repeatedly, in the primary's logs:

```
2021-11-13 21:12:00,631 ERROR [org.apache.activemq.artemis.core.client] AMQ214033: Cannot resolve host : java.net.UnknownHostException: brigade-artemis-secondary-0.brigade-artemis-secondary.brigade.svc.cluster.local
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) [rt.jar:1.8.0_312]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) [rt.jar:1.8.0_312]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) [rt.jar:1.8.0_312]
at java.net.InetAddress.getByName(InetAddress.java:1077) [rt.jar:1.8.0_312]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.isSameHostAndPort(NettyConnector.java:1291) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.isEquivalent(NettyConnector.java:1272) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.setBackupConnector(ClientSessionFactoryImpl.java:292) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1484) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$SessionFactoryTopologyHandler.notifyNodeUp(ClientSessionFactoryImpl.java:1497) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.notifyTopologyChange(ActiveMQClientProtocolManager.java:544) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager$Channel0Handler.handlePacket(ActiveMQClientProtocolManager.java:494) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:820) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:428) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:396) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1263) [artemis-core-client-2.19.0.jar:2.19.0]
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.19.0.jar:2.19.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.68.Final.jar:4.1.68.Final]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.19.0.jar:2.19.0]
```

I've seen failover work before. Not sure yet what's going on here.

Some experimentation may be in order.

Contributor guide

Open the contributing guide

Research direction

Start with the reported HA configuration and the primary logs, then trace the failure through NettyConnector.isSameHostAndPort, ClientSessionFactoryImpl.setBackupConnector, and ServerLocatorImpl.notifyNodeUp. Reproduce the Kubernetes hostname-resolution failure and determine whether the primary can resolve and fail over to the secondary. Done means the underlying cause is identified and primary-to-secondary visibility or failover works reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.