apache / apache/pulsar

[Bug] Producer creation gets blocked “Closed producer before its creation was completed”

Open
#22,217 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Pulsar 2.10.2

org.apache.pulsar
pulsar-client
2.10.2

### Minimal reproduce step

Call the Pulsar Client API to create a topic producer.

### What did you expect to see?

The client returns normally and the producer is created successfully.

### What did you see instead?

broker log:
1.Topic producer creating
`2024-03-05T02:35:13,559+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368][persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Creating producer. producerId=28`

2.It prompts that the creation is successful and then starts blocking. At this time, the client thread is also the same. The blocking process lasts 23 seconds.
`2024-03-05T02:35:13,591+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.broker.service.BrokerService - Created topic persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report - dedup is disabled`
`2024-03-05T02:35:36,644+0000 [pulsar-web-36-8] INFO org.eclipse.jetty.server.RequestLog - 192.168.64.2 - - [05/Mar/2024:02:35:36 +0000] "GET /admin/v2/clusters HTTP/1.1" 200 13 "-" "Pulsar-Java-v2.5.2" 1`

3.Client reports timeout error broker log printing
`2024-03-05T02:35:37,932+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368] Closed producer before its creation was completed. producerId=28`

4.The client initiates a request again to create the topic producer
`2024-03-05T02:35:37,953+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368][persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Creating producer. producerId=29`

5.The request creation is not blocked but returns an error
`2024-03-05T02:36:07,814+0000 [pulsar-io-4-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/42.193.247.132:55368] Closed producer before its creation was completed. producerId=29`

6.After continuing to request several times to create a topic, the broker prints an exception log:
`[persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Global topic inactive for 60 seconds, closed repl producers`
`2024-03-05T02:37:44,217+0000 [pulsar-inactivity-monitor-24-1] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Topic is already being closed or deleted`
`2024-03-05T02:37:44,217+0000 [pulsar-inactivity-monitor-24-1] WARN org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://rinoiot_1656861934759301120/SpwnFnVeq7QPbU2CNcbS/device_dp_report] Inactive topic deletion failed
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException: Topic is already fenced`

7.Subsequent requests to the topic kept reporting the error "Topic is already fenced", and finally restarted Pulsar to recover.

The following is the complete log of the broker:
![20240307151942](https://github.com/apache/pulsar/assets/47099845/bd90e4a3-720e-466e-9dec-f94ede54d639)
![20240307152153](https://github.com/apache/pulsar/assets/47099845/90b6d52f-206d-430b-854c-0b78d80aeec0)
![20240307152337](https://github.com/apache/pulsar/assets/47099845/25165a46-76ec-4c61-a8c0-1b6a2497ddf9)
![20240307152419](https://github.com/apache/pulsar/assets/47099845/18c69f52-8505-43e7-9f25-c86e86c4d3a2)

The following is my client-side producer code
![20240307152713](https://github.com/apache/pulsar/assets/47099845/77549ee8-8a51-4d20-91c1-24d0b6c95970)

### Anything else?

How can I improve my code or Pulsar configuration to avoid the blocking scenario, or allow operations to continue on a topic after it gets blocked? Once a topic enters the "fenced" state, the client is unable to perform any further operations on that topic besides restarting the broker.

Would the following information be helpful for my situation?
https://github.com/apache/pulsar/issues/20526#issuecomment-1581958101

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the producer-creation sequence against Pulsar 2.10.2 using the Pulsar Client API, then trace the broker log entries through ServerCnx, BrokerService, and PersistentTopic. Compare the client timeout with the subsequent fencing and inactivity-monitor messages; done means identifying the condition that blocks creation and leaves the topic fenced, with a reproducible case or confirmed configuration cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.