apache / apache/pulsar

[Doc] Add complete list of supported schemas and any transformations the connector completes before sending data to elastic

Open
#20,794 2 comments 0 reactions 0 assignees View on GitHub
doc-required Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### What issue do you find in Pulsar docs?

The docs for elastic search sink connector only mention Raw processing and Schema aware options for the connector. Raw processing says "The sink reads from topics and passes the raw contents to Elasticsearch." This somewhat implies sending any schema is possible, or maybe it means the schema must adhere to a strict format required by elastic search. It's unclear. In practice, it's currently mostly trial and error to find which schemas can be sent to elastic search and if Schema aware should be enabled. If there is a compatibility issue, you need to read the connector logs to determine the conversion problem.

### What is your suggestion?

Provide a comprehensive list of which schema types are supported with Raw processing and which schema types are supported with Schema aware. For each supported schema, include any transformation the connector may complete before sending the data to elastic search (e.g. is a string field wrapped with a data field to make a JSON compatible with elastic search?). This could be used as a reference guide for new users to quickly know if their schema is eligible for sending to elastic search before doing any trial and error (e.g. is a producer using Schema.STRING eligible versus a producer using Schema.JSON(String.class))?

### Any reference?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the Elasticsearch sink connector documentation, especially the Raw processing and Schema aware sections, and trace the connector behavior for each schema type. Document supported schemas, any transformations before sending data to Elasticsearch, and examples such as Schema.STRING versus Schema.JSON(String.class), so users can determine compatibility without trial and error.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.