confluentinc / confluentinc/examples
Make it work with Elasticsearch 7.1
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
Not really an issue, more some notes on
## How the clickstream example worked for me with Elasticsearch 7.1
### Modifications
* elastic-dynamic-template.sh: Elasticsearch template, remove `"_default_": {` and closing braces
* ksql-connect-es-grafana.sh: Grafana datasource, add `"esVersion":70` to jsonData
* ksql-connect-es-grafana.sh: Elasticsearch sinks, replace `"type.name": "type.name=kafkaconnect"` with `"type.name": "_doc"`. In Elasticsearch 7, they finally removed the types.
The `type.name` thing is a bit dirty, it would obviously be much better to have the Elasticsearch Kafka Connector work with Elasticsearch 7.
### If you have issues, remove everything
If you don't do those changes from the start, remove everything - indices and templates in Elasticsearch, sinks in Kafka, data sources in Grafana.
You may also want to restart the services.
### Put Elasticsearch template before the data comes
Also, I think that in `start.sh`, `elastic-dynamic-template.sh` should be called before `ksql-tables-to-grafana.sh`.
### Errors encountered
Errors that happened (adding this might make it easier to find for other users):
* If you didn't modify the Elasticsearch template, putting it will fail, and without template, Elasticsearch stores the epoch as a number and Grafana query fails with `Caused by: java.lang.RuntimeException: java.text.ParseException: Unparseable number:` error in the Elasticsearch logs
* without template, text fields aren't keywords, and some Grafana queries fail with `Set fielddata=true on [IP] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
` (visible in Grafana and Elasticsearch log)
* with template but the unmodified sink definition, documents cannot be indexed into Elasticsearch due to: `Rejecting mapping update to [errors_per_min] as the final mapping would have more than 1 type: [_doc, type.name=kafkaconnect` (Elasticsearch log)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.