[Enhancement] Support using sql to filter messages
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Currently, Pulsar uses the pluggable EntryFilter method for message filtering based on https://github.com/apache/pulsar/issues/12269. Users need to implement EntryFilter by themselves, and then filter messages on the broker side.
This way will require many tedious steps, as shown in the documentation https://github.com/apache/pulsar/blob/master/site2/docs/develop-plugin.md.
We hope to introduce a more convenient way to filter messages through sql. Consumers can implement automatic filtering on the broker side through simple subscriptions without implementing EntryFilter.
For example
We have the following message, which has two properties:
```
producer. newMessage()
.value(messageValue)
.property("country", "cn")
.property("age", "30")
.send();
```
On the consumer side, we want to filter messages through sql:
```
ConsumerBuilder builder = client. newConsumer(Schema. STRING)
.topic(topicName)
.subscriptionName(randomName(8))
.subscriptionType(SubscriptionType. Exclusive);
builder.subscriptBySql("country IS NOT NULL AND age IS NOT NULL AND country == 'cn' AND age >= 30 ")
```
### Solution
We need to implement a sql filtering mechanism on the broker side, that is, an EntryFilter that can support sql.
This way, users don't need to implement EntryFilter themselves.
In addition, we need to add an api similar to 'subscriptBySql' on the client side, so that users can subscribe more easily
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reading the existing EntryFilter approach and the develop-plugin.md documentation linked in the issue, then trace the client subscription builder API. Define how SQL expressions over message properties should be evaluated on the broker and how a corresponding client subscription method should behave. Done means broker-side SQL filtering works without a user-provided EntryFilter and consumers can configure it through the client API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100