apache / apache/pulsar

[Bug] can't create topic in k8s ipv6. broker

Open
#23,843 9 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

k8s with ipv6
imag version:3.3.3
no client, just cli.

### Minimal reproduce step

this is my broker deploy yaml file:
```yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: "my-release-pulsar-broker"
namespace: milvus
labels:
app: pulsar
chart: pulsar-2.7.8
release: my-release
heritage: Helm
cluster: my-release-pulsar
component: broker
spec:
serviceName: "my-release-pulsar-broker"
replicas: 3
selector:
matchLabels:
app: pulsar
release: my-release
component: broker
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: pulsar
release: my-release
cluster: my-release-pulsar
component: broker
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app: broker
component: bookie
topologyKey: kubernetes.io/hostname
weight: 1
serviceAccountName: "my-release-pulsar-broker-acct"
terminationGracePeriodSeconds: 30
initContainers:
- name: wait-zookeeper-ready
image: "registry.paas/eos-indexer/pulsar:3.3.3"
imagePullPolicy: IfNotPresent
command: ["sh", "-c"]
args:
- >-
until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server my-release-pulsar-zookeeper:2181 get /admin/clusters/my-release-pulsar; do
echo "pulsar cluster my-release-pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3;
done;
volumeMounts:
- name: wait-bookkeeper-ready
image: "registry.paas/eos-indexer/pulsar:3.3.3"
imagePullPolicy: IfNotPresent
command: ["sh", "-c"]
args:
- >
bin/apply-config-from-env.py conf/bookkeeper.conf;
until bin/bookkeeper shell whatisinstanceid; do
echo "bookkeeper cluster is not initialized yet. backoff for 3 seconds ...";
sleep 3;
done;
echo "bookkeeper cluster is already initialized";
bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)";
until [ ${bookieServiceNumber} -ge 2 ]; do
echo "bookkeeper cluster my-release-pulsar isn't ready yet ... check in 10 seconds ...";
sleep 10;
bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)";
done;
echo "bookkeeper cluster is ready";
envFrom:
- configMapRef:
name: "my-release-pulsar-bookie"
volumeMounts:
containers:
- name: "my-release-pulsar-broker"
image: "registry.paas/eos-indexer/pulsar:3.3.3"
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 1.5
memory: 4096Mi
command: ["sh", "-c"]
args:
- >
bin/apply-config-from-env.py conf/broker.conf;
bin/gen-yml-from-env.py conf/functions_worker.yml;
echo "OK" > status;
bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080;
while [ $? -eq 0 ]; do
echo "broker ${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local znode still exists ... check in 10 seconds ...";
sleep 10;
bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080;
done;
cat conf/pulsar_env.sh;
OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar broker;
ports:
- name: http
containerPort: 8080
hostPort: 8080
- name: "pulsar"
containerPort: 6650
hostPort: 6650
envFrom:
- configMapRef:
name: "my-release-pulsar-broker"
volumeMounts:
volumes:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: "my-release-pulsar-broker"
namespace: milvus
labels:
app: pulsar
chart: pulsar-2.7.8
release: my-release
heritage: Helm
cluster: my-release-pulsar
component: broker
data:
zookeeperServers: "my-release-pulsar-zookeeper:2181"
configurationStoreServers: "my-release-pulsar-zookeeper:2181"
clusterName: my-release-pulsar
exposeTopicLevelMetricsInPrometheus: "true"
numHttpServerThreads: "8"
zooKeeperSessionTimeoutMillis: "30000"
statusFilePath: "/pulsar/status"
functionsWorkerEnabled: "false"
webServicePort: "8080"
brokerServicePort: "6650"
PULSAR_GC: |
-Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError
PULSAR_MEM: |
-Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m
backlogQuotaDefaultLimitGB: "32"
backlogQuotaDefaultRetentionPolicy: producer_exception
defaultRetentionSizeInMB: "-1"
defaultRetentionTimeInMinutes: "10080"
managedLedgerDefaultAckQuorum: "1"
managedLedgerDefaultEnsembleSize: "1"
managedLedgerDefaultWriteQuorum: "1"
maxMessageSize: "104857600"
subscriptionExpirationTimeMinutes: "3"
ttlDurationDefaultInSeconds: "259200"
---
apiVersion: v1
kind: Service
metadata:
name: "my-release-pulsar-broker"
namespace: milvus
labels:
app: pulsar
chart: pulsar-2.7.8
release: my-release
heritage: Helm
cluster: my-release-pulsar
component: broker
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 8080
- name: "pulsar"
port: 6650
clusterIP: None
selector:
app: pulsar
release: my-release
component: broker
```

### What did you expect to see?

I am a newer for pulsar, I just want to run a demo of pulsar cluster with some node.
first. I deploy my pods in k8s cluster. they are look good.
then, I want to create a topic, but I'm failed.

### What did you see instead?

in pulsar-admin cli, I got this.
``` bash
my-release-pulsar-proxy-0:/pulsar# pulsar-admin topics create public/default/test1
2025-01-13T07:41:27,924+0000 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/default/test1] Failed to perform http put request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout
HTTP 504 Gateway Timeout

Reason: HTTP 504 Gateway Timeout
```
then I found some exception log in broker pod:
```
2025-01-13T07:41:04,652+0000 [pulsar-io-3-12] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 6.085 s
2025-01-13T07:41:10,739+0000 [broker-client-shared-timer-8-1] INFO org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Reconnecting after connection was closed
2025-01-13T07:41:10,742+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local/:6650 : io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://public/default/__change_events] lookup failed : java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
at org.apache.pulsar.client.impl.ConnectionPool.lambda$createConnection$13(ConnectionPool.java:333) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:408) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$18(ConnectionPool.java:409) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86) ~[io.netty-netty-resolver-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext$1.operationComplete(DnsResolveContext.java:268) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1158) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
... 4 more
Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1150) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
... 4 more
Caused by: io.netty.resolver.dns.DnsNameResolverException: [4836: /[fd11:1111:1112:15:0:0:0:a]:53] DefaultDnsQuestion(my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local. IN A) failed to send a query '4836' via UDP (no stack trace available)
Caused by: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol
2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
2025-01-13T07:41:10,744+0000 [pulsar-io-3-13] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 12.786 s
```
It appears that pulsar cannot use ipv6 dns server?
I confirm the dns server in log is working.
``` bash
my-release-pulsar-broker-0:/pulsar$ nslookup my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local fd11:1111:1112:15:0:0:0:a
Server: fd11:1111:1112:15:0:0:0:a
Address: fd11:1111:1112:15::a#53

Name: my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local
Address: 10.222.55.247
```
Can I run pulsar in ipv6 k8s cluster?

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the provided StatefulSet, ConfigMap, and headless Service YAML, then reproduce `pulsar-admin topics create public/default/test1` while checking DNS resolution for `my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local`. Done means the broker hostname resolves and topic creation completes without the HTTP 504 or connection errors shown in the broker logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.