Elasticsearch sink `keyIgnore` config support non-keyvalue schemas (i.e. avro and json)
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Is your enhancement request related to a problem? Please describe.**
Currently, `keyIgnore` config only works when keyvalue schemas are used for a topic, because of this line below
https://github.com/apache/pulsar/blob/ede95e9ba084c03822e40824c0ea41878f5fc102/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java#L169
**Describe the solution you'd like**
I'd like avro and json schemas to work as well when `keyIgnore = false`, since to my understanding, currently the golang and python client does not support keyvalue schema. So I can manually set the key as a primitive string, and still use this `keyIgnore` config to have `_id` in ES set as the message key.
I think the solution here is quite simple if I understand the code correctly here.
```
if (elasticSearchConfig.isKeyIgnore() == false && key != null) {
if (keySchema != null) {
// use key schema to encode the key
} else {
// use the key directly
}
}
```
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start in pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch/ElasticSearchSink.java at the referenced key-handling line. Trace how keyIgnore currently handles keyvalue schemas and how a direct message key is available for other schemas. Done means keyIgnore=false sets the Elasticsearch _id from primitive Avro or JSON message keys as well as keyvalue keys.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100