apache / apache/pulsar

Elasticsearch sink `keyIgnore` config support non-keyvalue schemas (i.e. avro and json)

Open
#13,173 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your enhancement request related to a problem? Please describe.**

Currently, `keyIgnore` config only works when keyvalue schemas are used for a topic, because of this line below

https://github.com/apache/pulsar/blob/ede95e9ba084c03822e40824c0ea41878f5fc102/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java#L169

**Describe the solution you'd like**

I'd like avro and json schemas to work as well when `keyIgnore = false`, since to my understanding, currently the golang and python client does not support keyvalue schema. So I can manually set the key as a primitive string, and still use this `keyIgnore` config to have `_id` in ES set as the message key.

I think the solution here is quite simple if I understand the code correctly here.
```
if (elasticSearchConfig.isKeyIgnore() == false && key != null) {
if (keySchema != null) {
// use key schema to encode the key
} else {
// use the key directly
}
}
```

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start in pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java at the referenced key-handling line. Trace how keyIgnore currently handles keyvalue schemas and how a direct message key is available for other schemas. Done means keyIgnore=false sets the Elasticsearch _id from primitive Avro or JSON message keys as well as keyvalue keys.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend, search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.