apache / apache/rocketmq

[Enhancement] Support Wildcard Pattern Matching for LiteTopic Consumer Subscriptions

Open
#10,559 2 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Enhancement Request

- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

### Summary

Support wildcard pattern matching for LiteTopic consumer subscriptions, enabling hierarchical topic organization and fine-grained subscriber-side filtering.

### Motivation

PR #10204 introduced Lite wildcard groups, but the current implementation has several limitations:

1. **No consumer-side pattern filtering** — A wildcard consumer group receives all LiteTopics under the parent topic. Consumers cannot express which subset they want; they always get everything.

2. **Full LMQ scan on every dispatch cycle** — `doFullDispatchForWildcardGroup` calls `forEachLiteTopic()` which scans the entire ConsumeQueue table (O(N) over all LMQs), not just the topics under the bound parent topic.

3. **No hierarchical topic organization** — LiteTopic names are flat strings with no way to express logical grouping (e.g., "all payment-related subtopics").

This makes it difficult to build systems where different consumers within the same group handle different subsets of events.

### Describe the Solution You'd Like

**Wildcard pattern matching** for LiteTopic consumer subscriptions, inspired by NATS subject matching.

#### Key Design Decisions

**Separator: `__` (double underscore)**
- `.` is already used in RocketMQ topic names (e.g., `%LMQ%$parent$child`) and in ConsumeQueue file paths
- `__` is visually distinctive, rarely appears in topic names, and avoids conflicts

**Pattern Tokens**

| Token | Meaning | Position | NATS Equivalent |
|-------|---------|----------|-----------------|
| `*` | Matches exactly one `__`-delimited segment | Anywhere | `*` |
| `**` | Matches one or more segments | Must be at the end | `>` |

**Examples** (parent topic = `order_events`)

| Pattern | Matches | Does Not Match |
|---------|---------|----------------|
| `pay__refund` | `pay__refund` | `pay__refund__notify`, `notify__refund` |
| `pay__*` | `pay__refund`, `pay__success` | `pay__refund__notify` |
| `*__refund` | `pay__refund`, `notify__refund` | `pay__refund__notify` |
| `pay__*__notify` | `pay__refund__notify`, `pay__success__notify` | `pay__refund` |
| `**` | `pay`, `pay__refund`, `pay__refund__notify` | (matches all) |
| `pay__**` | `pay__refund`, `pay__refund__notify` | `notify__refund` |

#### Implementation Plan

**Core matching engine:** `LitePatternMatcher` in `common` module, providing three static methods:
- `matches(pattern, liteTopic)` — recursive segment-by-segment matching, no regex
- `expand(pattern, candidates)` — expand a pattern against a set of candidate topics
- `validate(pattern)` — validate pattern syntax

**Two-phase subscription expansion:**
1. **Eager expansion** at `COMPLETE_ADD` time — expand all patterns against existing LiteTopics under the parent topic, register matched topics to `liteTopic2Group`
2. **Lazy re-expansion** when new LiteTopics are created via `onRegister()` — check if the new topic matches any client's patterns, add to subscription if so

**Dispatch optimization:** Replace full LMQ scan (`forEachLiteTopic`, O(N)) with targeted iteration via `collectByParentTopic` (O(M), where M = topics under the parent topic).

**Multi-pattern support (v1):** `liteTopicSet` is a `Set`, so multiple patterns can be sent in a single `COMPLETE_ADD` request. The union of all matched topics becomes the client's effective subscription.

**Protocol compatibility:** Reuse `LiteSubscriptionDTO.liteTopicSet` to carry pattern strings for wildcard groups. No wire protocol changes needed — the broker distinguishes via `isWildcardGroup()`.

#### Consumer API Example

```java
DefaultLitePullConsumer consumer = new DefaultLitePullConsumer("order_consumer_group");
consumer.setLiteBindTopic("order_events");
consumer.subscribeLite("pay__*"); // single-level payment events
consumer.subscribeLite("notify__**"); // all notification events (multi-level)
```

#### Key Changes

| Component | Change |
|-----------|--------|
| `LitePatternMatcher` (new) | Core matching engine in `common` |
| `LiteSubscription` | + `wildcardPatterns: Set` |
| `LiteSubscriptionRegistryImpl` | Multi-pattern expansion in `addCompleteSubscription` |
| `LiteEventDispatcher` | Lazy pattern re-expansion in `onRegister`; dispatch optimization |

### Describe Alternatives You've Considered

| Alternative | Why Rejected |
|-------------|-------------|
| Store compiled Trie per wildcard group | Over-engineering for v1; `String.split` + recursive matching is fast enough for < 10K topics |
| Support `**` in the middle of a pattern (e.g., `order__**__refund`) | NATS does not support this; greedy vs. non-greedy makes matching ambiguous; deferred to v2 |
| Use `.` as separator | Conflicts with existing topic naming conventions in RocketMQ |
| Per-message pattern matching (no eager expansion) | Would require checking patterns on every `doDispatch` call; eager expansion is more efficient |
| Add a dedicated `pattern` field to `LiteSubscriptionDTO` | Unnecessary; reusing `liteTopicSet` is simpler and avoids wire protocol changes |

### Additional Context

- Inspired by NATS subject-based addressing and MQTT topic filters (with `+` / `#`), but using `__` separator to avoid conflicts with RocketMQ's existing naming conventions
- Aligned with FlowMQ's "unified topic routing" philosophy — one topic namespace, multiple protocol bindings, unified wildcard matching
- Performance target: `matches()` < 5μs per call, `expand()` < 10ms for 1,000 candidates
- Full backward compatibility: wildcard groups without patterns continue to receive all LiteTopics under the parent topic (existing behavior)
- Registers via `LiteSubscriptionCtlRequestBody` without any wire protocol changes
- Reuses the existing `collectByParentTopic` API for dispatch optimization

Contributor guide

Open the contributing guide

Research direction

Start by reading the Lite wildcard-group implementation introduced in PR #10204, then trace LiteSubscriptionRegistryImpl.addCompleteSubscription and LiteEventDispatcher.onRegister. Review the planned LitePatternMatcher methods and the existing collectByParentTopic and forEachLiteTopic entry points. Done means validated multi-pattern subscriptions expand and re-expand correctly, dispatch scans only the parent topic, and existing unfiltered wildcard groups remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.