apache / apache/pulsar

[Enhancement] Support using sql to filter messages

Open
#18,483 7 comments 0 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

Currently, Pulsar uses the pluggable EntryFilter method for message filtering based on https://github.com/apache/pulsar/issues/12269. Users need to implement EntryFilter by themselves, and then filter messages on the broker side.
This way will require many tedious steps, as shown in the documentation https://github.com/apache/pulsar/blob/master/site2/docs/develop-plugin.md.

We hope to introduce a more convenient way to filter messages through sql. Consumers can implement automatic filtering on the broker side through simple subscriptions without implementing EntryFilter.

For example
We have the following message, which has two properties:
```
producer. newMessage()
.value(messageValue)
.property("country", "cn")
.property("age", "30")
.send();
```
On the consumer side, we want to filter messages through sql:
```
ConsumerBuilder builder = client. newConsumer(Schema. STRING)
.topic(topicName)
.subscriptionName(randomName(8))
.subscriptionType(SubscriptionType. Exclusive);

builder.subscriptBySql("country IS NOT NULL AND age IS NOT NULL AND country == 'cn' AND age >= 30 ")

```

### Solution

We need to implement a sql filtering mechanism on the broker side, that is, an EntryFilter that can support sql.
This way, users don't need to implement EntryFilter themselves.

In addition, we need to add an api similar to 'subscriptBySql' on the client side, so that users can subscribe more easily

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing EntryFilter approach and the develop-plugin.md documentation linked in the issue, then trace the client subscription builder API. Define how SQL expressions over message properties should be evaluated on the broker and how a corresponding client subscription method should behave. Done means broker-side SQL filtering works without a user-provided EntryFilter and consumers can configure it through the client API.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.