[Feature] The filtering feature of events
- Dominant language
- Java
- Stars
- 1.8k
- Forks
- 658
- Avg merge
- 11h 29m
- Merged PRs (30d)
- 52
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/eventmesh/issues?q=is%3Aissue) and found no similar issues.
### Feature Request
I would like to contribute a filtering feature for events. Here is my rough design idea.
First, when creating a subscription, add a new json String field 'filter' to the SubscriptionItem to describe the specific filtering rules.for example
```json
{
"and": [{
"field": "specversion",
"operator": "EQ",
"value": "1.0"
}, {
"or": [{
"field": "id",
"operator": "EQ",
"value": "9aeb0fdf-c01e-0131-0922-9eb54906e209"
}, {
"field": "email",
"operator": "CN",
"value": "example"
}]
}, {
"field": "data.contentType",
"operator": "SW",
"value": "image/png"
}]
}
```
When a consumer consumes, check the 'filter' field and convert it into a 'FilterGroup'. Then, generate a combined predicate based on the rules and provide the filtering results.
The logical relationship between the rules is described by 'and' and 'or'. The 'field' field describes the field to be checked, the 'operator' describes the specific rule, and the 'value' describes the comparison data for the rule.
The value of the field field can be a nested field in the event, expressed as a hierarchical relationship using '.'.
and This is the filtering process.
```mermaid
sequenceDiagram
participant Consumer
participant FilterGroup
participant SubscriptionItem
participant SubscriptionItem
Consumer->>+FilterGroup: Transform event to JSON
FilterGroup->>SubscriptionItem: Get filter rules JSON
FilterGroup->>FilterGroup: Parse filter rules JSON
FilterGroup->>FilterGroup: Generate combined predicate
FilterGroup->>+FilterRule: Predicate rules
Note over FilterRule: Loop for all rules
FilterRule->>FilterRule: Rule 1 apply
FilterRule->>FilterRule: Rule 2 apply
FilterRule->>FilterRule: Rule 3 apply
Note over FilterRule: End of loop
FilterRule->>-FilterGroup: Allowed or denied
alt FilterGroup allowed or denied
FilterGroup->>-Consumer: Allowed or denied
else FilterGroup throws exception
FilterGroup->>Consumer: Apply default strategy
end
```
And provide optional default policy to handle filtering exceptions for allowing or denying access.
I have initially designed the following specific types of rules:
| operator | class
| --- | --- |
| **EQ** | EqualFilterRule
| **NE** | NotEqualFilterRule
| **GT** | GreaterThanFilterRule
| **GE** | GreaterThanOrEqualFilterRule
| **LT** | LessThanFilterRule
| **LE** | LessThanOrEqualFilterRule
| BT | BetweenFilterRule
| **CN** | ContainsFilterRule
| NC | NotContainsFilterRule
| **SW** | StartsWithFilterRule |
| **EW** | EndsWithFilterRule
| **RE** | RegexFilterRule
| IN | InFilterRule
| NIN | NotInFilterRule
| EX | ExistsFilterRule
| NEX | NotExistsFilterRule
| NOT | NotFilterRule
| NL | NullFilterRule
| NN | NotNullFilterRule
| EQT | EpochTimeFilterRule
| GTT | GreaterThanTimeFilterRule
| LTT | LessThanTimeFilterRule
| BT | BetweenTimeFilterRule
I have started working on the implementation of this code. Please consider my approach, which can also be applied to similar transformation features.
The bolded rules have been implemented. Rules can be continuously extended by implementing the interface.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Research direction
The issue names SubscriptionItem, FilterGroup, and FilterRule but does not identify files, tests, or an entry point. Start by locating those types and the consumer subscription path, then verify how filtering should be integrated with event delivery. Done would require a reviewed filter schema, supported operators, nested-field handling, exception policy, and coverage for the proposed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100