[Bug] Proxy removes transaction subscriptions before the first producer heartbeat
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- java
- Domain
- distributed-systems, testing-qa
Research direction
Start with ClusterTransactionService.scanProducerHeartBeat(), ProducerManager.groupOnline(group), and the existing ClusterData entries. Reproduce the deterministic service-level sequence without a broker, then cover delayed first registration, never-registered expiration, renewed activity, replacement, online activity, and explicit unsubscription during the grace period. Done means recent subscriptions survive the first-heartbeat delay, offline groups avoid broker heartbeats, expired subscriptions are reclaimed, and registration resumes heartbeats.
Written by the indexing model from the issue text.
Description
Runtime platform environment
Proxy cluster mode with a Remoting transaction producer. The state transition can be reproduced deterministically in a service-level unit test, without a running broker or timing sleeps.
RocketMQ version
Current develop: 80e1ae55773c2330d2005b86f020ed028f649e94.
JDK Version
Local regression validation: Amazon Corretto 11.0.23, Maven 3.9.8, macOS arm64.
Describe the Bug
ClusterTransactionService.scanProducerHeartBeat() deletes a transaction subscription as soon as ProducerManager.groupOnline(group) returns false. A transaction send can create this subscription before the producer's first heartbeat registers its channel in the proxy.
If the scan runs in that interval, returning null from groupClusterData.computeIfPresent() removes the group-to-cluster mapping. A later producer heartbeat only updates ProducerManager; it does not rebuild the deleted transaction subscription. If the producer sends no further transaction messages, subsequent scans cannot register that group with the broker, so transaction checks cannot reach the producer.
This is a follow-up to #8316, which reports the same race and was closed by the stale bot on 2026-09-15. It remains reproducible in the current implementation. See also #5862 and its fix #5865: preventing indefinite heartbeats for producers that exit before their first heartbeat remains necessary.
The earlier fix proposal #8320 by @redlsz introduced a subscription timestamp and grace period for this race. It was closed as stale on 2026-05-18 without being merged, so those changes are not present in the current implementation.
Steps to Reproduce
Deterministic service-level sequence:
- Add a transaction subscription for a group and a topic with a valid broker-cluster route.
- Leave the group absent from
ProducerManager, representing a pending first producer heartbeat. - Run
scanProducerHeartBeat(). - Register the group, or change the mocked
groupOnline(group)result totrue. - Run
scanProducerHeartBeat()again, without adding another transaction subscription.
The group is removed at step 3. Step 5 has no subscription from which to build a broker heartbeat.
An end-to-end trigger is to send a transaction whose local result is UNKNOWN before the first producer heartbeat, keep the producer alive, and send no additional transaction messages. The failure depends on the scan falling between subscription creation and producer registration; it does not happen on every run.
What Did You Expect to See?
A recently created transaction subscription should survive a delayed first producer heartbeat and resume broker heartbeats once the producer becomes online.
For the #5862 case, offline groups should receive no broker heartbeats, and subscriptions for producers that never register should eventually be reclaimed. Explicit group unregistration should still remove subscriptions immediately.
What Did You See Instead?
The subscription is removed immediately while the group is not yet registered. Registration alone does not restore it. The broker can then fail to find a channel for transaction checks, and the producer's transaction checker is not invoked.
Proposed Fix
Keep a millisecond lastActiveTimestamp, using System.currentTimeMillis(), on the existing ClusterData entries. Refresh it when subscriptions are added/replaced and while the group is observed online. When the group is offline, skip broker heartbeats immediately and expire subscriptions only after channelExpiredTimeout (currently 120 seconds by default). Preserve the existing explicit-unsubscribe path.
Regression coverage should include delayed first registration, never-registered expiration, renewed subscription activity, online activity, replacement, and explicit unsubscription during the grace period.
- Dominant language
- Java
- Stars
- 22.6k
- Forks
- 12k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 24
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/rocketmq
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
[Bug] TopicValidator rejects V2 retry topic names containing `+` separator introduced by KeyBuilder Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 1/5 Under an hour Newbie friendliness 94/100
objectionary/hone-maven-plugin#1061 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
spring-projects/spring-modulith#1895 ·