apache / apache/beam

A Complex Event Processing (CEP) library/extension for Apache Beam

Open
#18,730 0 comments 2 reactions 0 assignees View on GitHub
gsoc mentor new feature P3 sdk-ideas
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Apache Beam [1] is a unified and portable programming model for data processing jobs. The Beam model [2, 3, 4] has rich mechanisms to process endless streams of events.

Complex Event Processing [5] lets you match patterns of events in streams to detect important patterns in data and react to them.

Some examples of uses of CEP are fraud detection for example by detecting unusual behavior (patterns of activity), e.g. network intrusion, suspicious banking transactions, etc. Also trend detection is another interesting use case in the context of sensors and IoT.

The goal of this issue is to implement an efficient pattern matching library inspired by [6] and existing libraries like Apache Flink CEP [7] using the Apache Beam Java SDK and the Beam style guides [8]. Because of the time constraints of GSoC we will probably try to cover first simple patterns of the ‘a followed by b followed by c’ kind, and then if there is still time try to cover more advanced ones e.g. optional, atLeastOne, oneOrMore, etc.

[1] [https://beam.apache.org/](https://beam.apache.org/)
[2] [https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101)
[3] [https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102)
[4] [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf)
[5] [https://en.wikipedia.org/wiki/Complex_event_processing](https://en.wikipedia.org/wiki/Complex_event_processing)
[6] [https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf](https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf)
[7] [https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html](https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html)
[8] [https://beam.apache.org/contribute/ptransform-style-guide/](https://beam.apache.org/contribute/ptransform-style-guide/)

 

Imported from Jira [BEAM-3767](https://issues.apache.org/jira/browse/BEAM-3767). Original Jira may contain additional context.
Reported by: iemejia.

Contributor guide

Open the contributing guide

Research direction

No source file or test is identified. Start by reviewing the Apache Beam Java SDK and the Beam PTransform style guide, then compare the referenced SASE and Flink CEP designs. Done would require an efficient library for simple sequential event patterns such as a followed by b followed by c; optional and repetition patterns are possible later.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.