apache / apache/pulsar

support namespace for index templating on elasticsearch sink connector

Open
#20,771 2 comments 0 reactions 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

I use apache pulsar for moving various logs such as infra, system, service, and other logs. I split them using pulsar namespace, and search in elasticsearch. but when using es sink connector, topics split by namespace are re-merged by topic name because we cannot use namespace for templating.

to avoid that currently I putting namespace name in topic for es, but I think this is quite disturbing.

### Solution

https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/IndexNameFormatter.java

support namespace substitution along with time partitioning.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/IndexNameFormatter.java and read how time partitioning is currently handled. Define how namespace substitution should work alongside that behavior, then verify that namespace-separated topics no longer re-merge by topic name in Elasticsearch.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
search, stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.