apache / apache/eventmesh

[Feature] The filtering feature of events

Open
#3,263 4 comments 0 reactions 0 assignees View on GitHub
feature WIP
Dominant language
Java
Stars
1.8k
Forks
658
Avg merge
11h 29m
Merged PRs (30d)
52

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/eventmesh/issues?q=is%3Aissue) and found no similar issues.

### Feature Request

I would like to contribute a filtering feature for events. Here is my rough design idea.
First, when creating a subscription, add a new json String field 'filter' to the SubscriptionItem to describe the specific filtering rules.for example
```json
{
"and": [{
"field": "specversion",
"operator": "EQ",
"value": "1.0"
}, {
"or": [{
"field": "id",
"operator": "EQ",
"value": "9aeb0fdf-c01e-0131-0922-9eb54906e209"
}, {
"field": "email",
"operator": "CN",
"value": "example"
}]
}, {
"field": "data.contentType",
"operator": "SW",
"value": "image/png"
}]
}
```
When a consumer consumes, check the 'filter' field and convert it into a 'FilterGroup'. Then, generate a combined predicate based on the rules and provide the filtering results.
The logical relationship between the rules is described by 'and' and 'or'. The 'field' field describes the field to be checked, the 'operator' describes the specific rule, and the 'value' describes the comparison data for the rule.
The value of the field field can be a nested field in the event, expressed as a hierarchical relationship using '.'.
and This is the filtering process.
```mermaid
sequenceDiagram
participant Consumer
participant FilterGroup
participant SubscriptionItem
participant SubscriptionItem
Consumer->>+FilterGroup: Transform event to JSON
FilterGroup->>SubscriptionItem: Get filter rules JSON
FilterGroup->>FilterGroup: Parse filter rules JSON
FilterGroup->>FilterGroup: Generate combined predicate
FilterGroup->>+FilterRule: Predicate rules
Note over FilterRule: Loop for all rules
FilterRule->>FilterRule: Rule 1 apply
FilterRule->>FilterRule: Rule 2 apply
FilterRule->>FilterRule: Rule 3 apply
Note over FilterRule: End of loop
FilterRule->>-FilterGroup: Allowed or denied
alt FilterGroup allowed or denied
FilterGroup->>-Consumer: Allowed or denied
else FilterGroup throws exception
FilterGroup->>Consumer: Apply default strategy
end
```
And provide optional default policy to handle filtering exceptions for allowing or denying access.
I have initially designed the following specific types of rules:
| operator | class
| --- | --- |
| **EQ** | EqualFilterRule
| **NE** | NotEqualFilterRule
| **GT** | GreaterThanFilterRule
| **GE** | GreaterThanOrEqualFilterRule
| **LT** | LessThanFilterRule
| **LE** | LessThanOrEqualFilterRule
| BT | BetweenFilterRule
| **CN** | ContainsFilterRule
| NC | NotContainsFilterRule
| **SW** | StartsWithFilterRule |
| **EW** | EndsWithFilterRule
| **RE** | RegexFilterRule
| IN | InFilterRule
| NIN | NotInFilterRule
| EX | ExistsFilterRule
| NEX | NotExistsFilterRule
| NOT | NotFilterRule
| NL | NullFilterRule
| NN | NotNullFilterRule
| EQT | EpochTimeFilterRule
| GTT | GreaterThanTimeFilterRule
| LTT | LessThanTimeFilterRule
| BT | BetweenTimeFilterRule

I have started working on the implementation of this code. Please consider my approach, which can also be applied to similar transformation features.
The bolded rules have been implemented. Rules can be continuously extended by implementing the interface.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

The issue names SubscriptionItem, FilterGroup, and FilterRule but does not identify files, tests, or an entry point. Start by locating those types and the consumer subscription path, then verify how filtering should be integrated with event delivery. Done would require a reviewed filter schema, supported operators, nested-field handling, exception policy, and coverage for the proposed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.