apache / apache/beam

[Parent issue] Support for Apache Pulsar

Open
#31,078 3 comments 3 reactions 0 assignees View on GitHub
P3 pinned task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Since there is a continious demand for it,
just an overview for current state of Support for Apache Pulsar:

- when looking into beams i/o connector matrix **pulsar is not listed yet**:
https://beam.apache.org/documentation/io/connectors/

- but there is a **sdk package**
Package org.apache.beam.sdk.io.pulsar
https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html

- in v2.39 of beam a relatively **basic pulsar io** was introduced:
Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/
for details see:
https://issues.apache.org/jira/browse/BEAM-8218

- there are some **open issues** in beam for pulsar:
https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen

- #30688
- #30152
- #22054
- #22027
- #22025
- #21714
- #21560
- #31079

- there were also some non merged pull request marked as stale and **automatically closed, which may be worth to have a look on**
https://github.com/apache/beam/pulls?q=is%3Apr+pulsar+is%3Aclosed+label%3Astale

- #26410
- #22375
- #22028
- #22026
- #17473
- #16749

- in general it would be a good longterm target, to reach the same quality of integration in Beam as there is for Kafka

Contributor guide

Open the contributing guide

Research direction

Start with the Apache Pulsar SDK package and the Beam I/O connector matrix, then review the linked open issues (#30688, #30152, #22054, #22027, #22025, #21714, #21560, and #31079). The stale closed pull requests may provide additional context. Done would mean defining and implementing a Pulsar integration with quality comparable to Kafka.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.