[Doc] Add complete list of supported schemas and any transformations the connector completes before sending data to elastic
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### What issue do you find in Pulsar docs?
The docs for elastic search sink connector only mention Raw processing and Schema aware options for the connector. Raw processing says "The sink reads from topics and passes the raw contents to Elasticsearch." This somewhat implies sending any schema is possible, or maybe it means the schema must adhere to a strict format required by elastic search. It's unclear. In practice, it's currently mostly trial and error to find which schemas can be sent to elastic search and if Schema aware should be enabled. If there is a compatibility issue, you need to read the connector logs to determine the conversion problem.
### What is your suggestion?
Provide a comprehensive list of which schema types are supported with Raw processing and which schema types are supported with Schema aware. For each supported schema, include any transformation the connector may complete before sending the data to elastic search (e.g. is a string field wrapped with a data field to make a JSON compatible with elastic search?). This could be used as a reference guide for new users to quickly know if their schema is eligible for sending to elastic search before doing any trial and error (e.g. is a producer using Schema.STRING eligible versus a producer using Schema.JSON(String.class))?
### Any reference?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with the Elasticsearch sink connector documentation, especially the Raw processing and Schema aware sections, and trace the connector behavior for each schema type. Document supported schemas, any transformations before sending data to Elasticsearch, and examples such as Schema.STRING versus Schema.JSON(String.class), so users can determine compatibility without trial and error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100