apache / apache/rocketmq-clients

[Bug][Go] Producer.Start() deadlocks infinitely when no topics are pre-registered (WithTopics not used) — introduced in v5.1.4

Open
#1,316 13 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
505
Forks
313
Avg merge
11h 28m
Merged PRs (30d)
6

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions).

- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq-clients/issues) and [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions) of this repository and believe that this is not a duplicate.

- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Programming Language of the Client

Go

### Runtime Platform Environment

- SDK version: v5.1.4 (regression from v5.1.1-rc1)
- Language: Go
- Component: Producer (defaultProducer / defaultClient)
- Broker: Apache RocketMQ 5.x

### RocketMQ Version of the Client/Server

Comparison: v5.1.1-rc1 vs v5.1.4
v5.1.1-rc1:
- startUp() return:Immediate after ticker starts
- defaultClient has inited field:No
- Behavior with empty initTopics:Works normally (lazy route fetch on first Send)
- Telemetry stream lifecycle:Created lazily on first getMessageQueues call

v5.1.4:
- startUp() return:Blocks until inited=true
- defaultClient has inited field:Yes (atomic.Bool)
- Behavior with empty initTopics:Deadlocks forever
- Telemetry stream lifecycle:Same, but startUp waits for it synchronously

v5.1.1-rc1 startUp() tail (lines 493–549):
ticker.Tick(f, time.Second*30, cli.done)
return nil // ← returns immediately, no blocking

```go
v5.1.4 startUp() tail (lines 596–607):
ticker.Tick(f, time.Second*30, cli.done)

// wait syncSettings finish ← NEW, BLOCKING
for !cli.inited.Load() {
if cli.startUpError != nil {
return cli.startUpError
}
sugarBaseLogger.Infoln("wait for sync settings finish")
time.Sleep(time.Second)
}

### Run or Compiler Version

_No response_

### Describe the Bug

In **v5.1.4**, `Producer.Start()` blocks forever when the producer is created
without `WithTopics(...)`. The root cause is a new blocking wait loop added to
`defaultClient.startUp()` that waits for `cli.inited` to become `true`. However,
`cli.inited` is only set by `onSettingsCommand()`, which can only be triggered
after a Telemetry stream is established with a broker — and that only happens
inside `getMessageQueues()`, which is only called when `initTopics` is non-empty.

When `initTopics` is empty (the default, and the most common usage pattern),
the loop spins forever:

```go
// client.go:599-606 (v5.1.4)
for !cli.inited.Load() { // inited stays false forever
if cli.startUpError != nil {
return cli.startUpError
}
sugarBaseLogger.Infoln("wait for sync settings finish")
time.Sleep(time.Second) // ← goroutine blocked here indefinitely
}

This behavior was not present in v5.1.1-rc1, where startUp() returned
immediately after starting the route-refresh ticker.

### Steps to Reproduce

```go
producer, _ := golang.NewProducer(
&golang.Config{
Endpoint: "...",
NameSpace: "...",
Credentials: &credentials.SessionCredentials{...},
},
golang.WithMaxAttempts(3),
// NOTE: WithTopics(...) intentionally omitted — this is the standard usage
// when topics are only known at send time
)

err := producer.Start() // ← blocks forever, never returns

### What Did You Expect to See?

Design Intent vs. Actual Behavior

The intent of the new blocking wait is understandable: ensure the producer has
received broker settings (retry policy, message-type validation, etc.) before
Start() returns, so the first Send() call is guaranteed to operate with a
fully configured client.

However, the implementation has a fundamental coupling problem:

- cli.inited depends on onSettingsCommand()
- onSettingsCommand() depends on a Telemetry stream being open
- A Telemetry stream is only opened via getMessageQueues()
- getMessageQueues() requires a topic to query
- Topics are only available at startup if the user passed WithTopics(...)

The Settings handshake is a client-level concern (it applies to the producer
as a whole), but the implementation ties it to a topic-level operation. These
two concerns should not be coupled.

In the existing API contract, WithTopics() is documented as an optional
pre-warming hint for route prefetching — it was never intended to be a
prerequisite for Start() to succeed.

### What Did You See Instead?

Evidence: goroutine dump

From a production goroutine dump collected while the service was completely
unresponsive:

```go
The goroutine stuck in the infinite wait loop:
goroutine 49529 [sleep]:
time.Sleep(0x3b9aca00)
.../runtime/time.go:363
github.com/apache/rocketmq-clients/golang/v5.(*defaultClient).startUp(0xc002893080)
.../rocketmq-clients/golang/v5@v5.1.4/client.go:604 ← inside the loop
github.com/apache/rocketmq-clients/golang/v5.(*defaultProducer).Start(0xc00799a4d0)
.../rocketmq-clients/golang/v5@v5.1.4/producer.go:56

Hundreds of request goroutines piled up waiting for the mutex:
goroutine 511012 [sync.Mutex.Lock, 1 minutes]:
...AcquireRocketMqProducer (producer.go:47)

goroutine 183826 [sync.Mutex.Lock, 3 minutes]:
...AcquireRocketMqProducer (producer.go:47)

goroutine 305925 [sync.Mutex.Lock, 2 minutes]:
...AcquireRocketMqProducer (producer.go:47)

goroutine 679238 [sync.Mutex.Lock]:
...AcquireRocketMqProducer (producer.go:47)

All of them trace back to the same service handler, proving a complete
send-path blockage caused solely by Start() never returning.

### Additional Context

Proposed Fix

Option A — Skip the wait when initTopics is empty (minimal, non-breaking)

```go
// client.go, end of startUp()
ticker.Tick(f, time.Second*30, cli.done)

// Only wait for Settings if topics were pre-registered.
// When initTopics is empty there is no Telemetry stream yet;
// the stream (and Settings) will be established lazily on first Send.
if len(cli.initTopics) == 0 {
return nil
}

for !cli.inited.Load() {
if cli.startUpError != nil {
return cli.startUpError
}
sugarBaseLogger.Infoln("wait for sync settings finish")
time.Sleep(time.Second)
}
```

Option B — Establish a Telemetry stream unconditionally at startup (more robust)

```go
Initiate a Telemetry connection to cli.accessPoint directly in startUp(),
independent of topic route queries. This removes the coupling entirely and makes
the Settings handshake reliable regardless of whether WithTopics was used.
```

Option C — Add a timeout to the wait loop (safety net, not a full fix)

```go
deadline := time.Now().Add(30 * time.Second)
for !cli.inited.Load() {
if cli.startUpError != nil {
return cli.startUpError
}
if time.Now().After(deadline) {
return fmt.Errorf("timed out waiting for broker Settings after 30s; " +
"no topics pre-registered via WithTopics, Telemetry stream not established")
}
time.Sleep(time.Second)
}
```
This at minimum turns an infinite hang into a fast-fail with a clear error
message.

---
Workaround (for users on v5.1.4 today)

Pass at least one known business topic via WithTopics(...) when creating the
producer. A single topic is sufficient to trigger the Telemetry handshake; other
topics used in Send() calls continue to work via the existing lazy-fetch path:

```go
producer, _ := golang.NewProducer(
config,
golang.WithMaxAttempts(3),
golang.WithTopics("your_topic"), // ← unblocks Start()
)
```

Note: this is a workaround, not the correct long-term fix. WithTopics should
remain optional as it was before v5.1.4.

---
Related

- defaultClient.startUp(): client.go:540–608
- defaultClient.inited field: client.go:239
- onSettingsCommand() sets inited: client.go:726
- defaultClientSession.startUp() propagates error: client.go:122–125
- WithTopics() option: producer_options.go:78

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with client.go:540–608, especially defaultClient.startUp(), then trace defaultClient.inited at client.go:239, onSettingsCommand() at client.go:726, and WithTopics() in producer_options.go:78. Reproduce the example with no pre-registered topics and verify that Producer.Start() returns without requiring WithTopics while preserving the settings and lazy route behavior described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.