[Parent issue] Support for Apache Pulsar
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Since there is a continious demand for it,
just an overview for current state of Support for Apache Pulsar:
- when looking into beams i/o connector matrix **pulsar is not listed yet**:
https://beam.apache.org/documentation/io/connectors/
- but there is a **sdk package**
Package org.apache.beam.sdk.io.pulsar
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html
- in v2.39 of beam a relatively **basic pulsar io** was introduced:
Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/
for details see:
https://issues.apache.org/jira/browse/BEAM-8218
- there are some **open issues** in beam for pulsar:
https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen
- #30688
- #30152
- #22054
- #22027
- #22025
- #21714
- #21560
- #31079
- there were also some non merged pull request marked as stale and **automatically closed, which may be worth to have a look on**
https://github.com/apache/beam/pulls?q=is%3Apr+pulsar+is%3Aclosed+label%3Astale
- #26410
- #22375
- #22028
- #22026
- #17473
- #16749
- in general it would be a good longterm target, to reach the same quality of integration in Beam as there is for Kafka
Contributor guide
Research direction
Start with the Apache Pulsar SDK package and the Beam I/O connector matrix, then review the linked open issues (#30688, #30152, #22054, #22027, #22025, #21714, #21560, and #31079). The stale closed pull requests may provide additional context. Done would mean defining and implementing a Pulsar integration with quality comparable to Kafka.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100