support namespace for index templating on elasticsearch sink connector
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
I use apache pulsar for moving various logs such as infra, system, service, and other logs. I split them using pulsar namespace, and search in elasticsearch. but when using es sink connector, topics split by namespace are re-merged by topic name because we cannot use namespace for templating.
to avoid that currently I putting namespace name in topic for es, but I think this is quite disturbing.
### Solution
https://github.com/apache/pulsar/blob/master/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/IndexNameFormatter.java
support namespace substitution along with time partitioning.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Research direction
Start with pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/IndexNameFormatter.java and read how time partitioning is currently handled. Define how namespace substitution should work alongside that behavior, then verify that namespace-separated topics no longer re-merge by topic name in Elasticsearch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- search, stream-processing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100