[Bug] Proxy removes transaction subscriptions before the first producer heartbeat

Open
#11,170 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
java

Research direction

Start with ClusterTransactionService.scanProducerHeartBeat(), ProducerManager.groupOnline(group), and the existing ClusterData entries. Reproduce the deterministic service-level sequence without a broker, then cover delayed first registration, never-registered expiration, renewed activity, replacement, online activity, and explicit unsubscription during the grace period. Done means recent subscriptions survive the first-heartbeat delay, offline groups avoid broker heartbeats, expired subscriptions are reclaimed, and registration resumes heartbeats.

Written by the indexing model from the issue text.

Description

Runtime platform environment

Proxy cluster mode with a Remoting transaction producer. The state transition can be reproduced deterministically in a service-level unit test, without a running broker or timing sleeps.

RocketMQ version

Current develop: 80e1ae55773c2330d2005b86f020ed028f649e94.

JDK Version

Local regression validation: Amazon Corretto 11.0.23, Maven 3.9.8, macOS arm64.

Describe the Bug

ClusterTransactionService.scanProducerHeartBeat() deletes a transaction subscription as soon as ProducerManager.groupOnline(group) returns false. A transaction send can create this subscription before the producer's first heartbeat registers its channel in the proxy.

If the scan runs in that interval, returning null from groupClusterData.computeIfPresent() removes the group-to-cluster mapping. A later producer heartbeat only updates ProducerManager; it does not rebuild the deleted transaction subscription. If the producer sends no further transaction messages, subsequent scans cannot register that group with the broker, so transaction checks cannot reach the producer.

This is a follow-up to #8316, which reports the same race and was closed by the stale bot on 2026-09-15. It remains reproducible in the current implementation. See also #5862 and its fix #5865: preventing indefinite heartbeats for producers that exit before their first heartbeat remains necessary.

The earlier fix proposal #8320 by @redlsz introduced a subscription timestamp and grace period for this race. It was closed as stale on 2026-05-18 without being merged, so those changes are not present in the current implementation.

Steps to Reproduce

Deterministic service-level sequence:

  1. Add a transaction subscription for a group and a topic with a valid broker-cluster route.
  2. Leave the group absent from ProducerManager, representing a pending first producer heartbeat.
  3. Run scanProducerHeartBeat().
  4. Register the group, or change the mocked groupOnline(group) result to true.
  5. Run scanProducerHeartBeat() again, without adding another transaction subscription.

The group is removed at step 3. Step 5 has no subscription from which to build a broker heartbeat.

An end-to-end trigger is to send a transaction whose local result is UNKNOWN before the first producer heartbeat, keep the producer alive, and send no additional transaction messages. The failure depends on the scan falling between subscription creation and producer registration; it does not happen on every run.

What Did You Expect to See?

A recently created transaction subscription should survive a delayed first producer heartbeat and resume broker heartbeats once the producer becomes online.

For the #5862 case, offline groups should receive no broker heartbeats, and subscriptions for producers that never register should eventually be reclaimed. Explicit group unregistration should still remove subscriptions immediately.

What Did You See Instead?

The subscription is removed immediately while the group is not yet registered. Registration alone does not restore it. The broker can then fail to find a channel for transaction checks, and the producer's transaction checker is not invoked.

Proposed Fix

Keep a millisecond lastActiveTimestamp, using System.currentTimeMillis(), on the existing ClusterData entries. Refresh it when subscriptions are added/replaced and while the group is observed online. When the group is offline, skip broker heartbeats immediately and expire subscriptions only after channelExpiredTimeout (currently 120 seconds by default). Preserve the existing explicit-unsubscribe path.

Regression coverage should include delayed first registration, never-registered expiration, renewed subscription activity, online activity, replacement, and explicit unsubscription during the grace period.

Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 12h
Merged PRs (30d)
24

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/rocketmq

All issues in apache/rocketmq

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.