confluentinc / confluentinc/examples

Make it work with Elasticsearch 7.1

Open
#173 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Not really an issue, more some notes on

## How the clickstream example worked for me with Elasticsearch 7.1
### Modifications
* elastic-dynamic-template.sh: Elasticsearch template, remove `"_default_": {` and closing braces
* ksql-connect-es-grafana.sh: Grafana datasource, add `"esVersion":70` to jsonData
* ksql-connect-es-grafana.sh: Elasticsearch sinks, replace `"type.name": "type.name=kafkaconnect"` with `"type.name": "_doc"`. In Elasticsearch 7, they finally removed the types.

The `type.name` thing is a bit dirty, it would obviously be much better to have the Elasticsearch Kafka Connector work with Elasticsearch 7.

### If you have issues, remove everything
If you don't do those changes from the start, remove everything - indices and templates in Elasticsearch, sinks in Kafka, data sources in Grafana.
You may also want to restart the services.

### Put Elasticsearch template before the data comes
Also, I think that in `start.sh`, `elastic-dynamic-template.sh` should be called before `ksql-tables-to-grafana.sh`.

### Errors encountered
Errors that happened (adding this might make it easier to find for other users):
* If you didn't modify the Elasticsearch template, putting it will fail, and without template, Elasticsearch stores the epoch as a number and Grafana query fails with `Caused by: java.lang.RuntimeException: java.text.ParseException: Unparseable number:` error in the Elasticsearch logs
* without template, text fields aren't keywords, and some Grafana queries fail with `Set fielddata=true on [IP] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
` (visible in Grafana and Elasticsearch log)
* with template but the unmodified sink definition, documents cannot be indexed into Elasticsearch due to: `Rejecting mapping update to [errors_per_min] as the final mapping would have more than 1 type: [_doc, type.name=kafkaconnect` (Elasticsearch log)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.