apache / apache/pulsar

Add ElasticSearch BatchSource

Open
#20,890 3 comments 0 reactions 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

ElasticSearch and Pulsar are very different types of data storage systems, however, there may be occasions where large amount of data that had been previously stored in ElasticSearch needs to be read into a Pulsar topic. We are working on transitioning a large data storage and search system which has tens of billions of records and hundreds of TB of data that would better serve the organization in Pulsar. There are likely other use cases where data is being ingested to ElasticSearch but also needs to be loaded into Pulsar at scale. The goal of this is not change detection, but simply to run a query and stream the results to a topic in Pulsar in a highly parallel way that will allow for copying very large datasets to Pulsar topics.

### Solution

Implement a BatchSource that is able to leverage the existing ElasticSearchSink code to build an ElasticSearch client, conduct a search that can be parallelized and send the results to a Pulsar topic. The motivation behind using the BatchSource is to use the `discover` phase to create and distribute the search slices, allowing the user to create a sliced search to be executed in parallel with multiple function runners. The user will be able to choose to run a Scroll or PIT search depending on what their ElasticSearch version supports.

### Alternatives

_No response_

### Anything else?

I'm working on the PR implementing this feature.

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing ElasticSearchSink code and the BatchSource discover phase used to distribute work across function runners. Trace how sliced Scroll or PIT searches would send results to a Pulsar topic; done means a parallel BatchSource supports the selected search mode and large-scale data transfer.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.