apache / apache/pulsar-client-go

The producer is gone, after modifying the parameter defaultNumberOfNamespaceBundles and restarting the brokers

Open
#189 2 comments 0 reactions 0 assignees View on GitHub
triage/week-15
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

#### Expected behavior

The application can continue to work without being affected

#### Actual behavior

the application cannot produce messages to the broker

#### Steps to reproduce

Occasional

#### System configuration
**Pulsar version**: 2.4.0
There are 3 brokers in our Pulsar cluster。

#### What happened
The parameter defaultNumberOfNamespaceBundles was set 4. After Modify parameter defaultNumberOfNamespaceBundles to 6 and restarting the brokers, the producers of one topic were gone. Every partition topic should have two producers.

before restarting the brokers :
time="2020-02-17T05:00:29+08:00" level=debug msg="Lookup result: &{LogicalAddr:pulsar://10.32.69.26:6651 PhysicalAddr:pulsar://10.32.69.26:6651}" name=qtvhr subscription=worker_1v1 topic=2/signal_geo/redis_sync-partition-0
time="2020-02-17T04:59:56+08:00" level=debug msg="Lookup result: &{LogicalAddr:pulsar://10.32.69.26:6650 PhysicalAddr:pulsar://10.32.69.26:6650}" name=qtvhr subscription=worker_1v1 topic=2/signal_geo/redis_sync-partition-1

after restarting the brokers:
[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics lookup persistent://2/signal_geo/redis_sync-partition-0
"pulsar://10.32.69.26:6650"
[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics lookup persistent://2/signal_geo/redis_sync-partition-1
"pulsar://10.32.69.26:6652"

[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics stats persistent://2/signal_geo/redis_sync-partition-0
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 103,
"publishers" : [ ],
"subscriptions" : {
"worker_1v1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "auurc",
"availablePermits" : 2048,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"address" : "/10.32.32.223:41824",
"connectedSince" : "2020-02-21T20:25:16.856+08:00"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "lnmnh",
"availablePermits" : 2048,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"address" : "/10.32.68.186:30702",
"connectedSince" : "2020-02-21T20:25:16.857+08:00"
} ],
"isReplicated" : false
}
},
"replication" : {
"qa-pulsar-ten" : {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 0,
"connected" : true,
"replicationDelayInSeconds" : 0,
"inboundConnection" : "/10.32.32.20:5644",
"inboundConnectedSince" : "2020-02-24T18:54:16.203+08:00",
"outboundConnection" : "[id: 0x4fd18011, L:/10.32.69.26:34288 - R:10.32.32.20/10.32.32.20:6651]",
"outboundConnectedSince" : "2020-02-24T18:54:16.257+08:00"
}
},
"deduplicationStatus" : "Disabled"
}

[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics stats persistent://2/signal_geo/redis_sync-partition-1
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 590,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"producerId" : 3,
"metadata" : { },
"producerName" : "qa-pulsar-ali-29-4286",
"connectedSince" : "2020-02-21T20:24:43.622+08:00",
"address" : "/10.32.68.186:43841"
} ],
"subscriptions" : {
"worker_1v1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "lnmnh",
"availablePermits" : 3061,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2020-02-21T20:25:16.647+08:00",
"address" : "/10.32.68.186:43841"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "auurc",
"availablePermits" : 3072,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2020-02-21T20:25:16.751+08:00",
"address" : "/10.32.32.223:49520"
} ],
"isReplicated" : false
}
},
"replication" : {
"qa-pulsar-ten" : {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 0,
"connected" : true,
"replicationDelayInSeconds" : 0,
"inboundConnection" : "/10.32.32.20:7870",
"inboundConnectedSince" : "2020-02-24T19:40:16.119+08:00",
"outboundConnection" : "[id: 0xbba2928c, L:/10.32.69.26:4291 - R:10.32.32.20/10.32.32.20:6652]",
"outboundConnectedSince" : "2020-02-24T19:40:16.153+08:00"
}
},
"deduplicationStatus" : "Disabled"
}

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reported Pulsar 2.4.0 configuration change from defaultNumberOfNamespaceBundles=4 to 6 and comparing the provided topic lookup and stats output before and after broker restarts. Reproduce the producer loss if possible and identify why one partition topic has no publishers; done means producers reconnect and can publish after the restart without affecting the application.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.