Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Unable to push data from one MSK topic into two OpenSearch indices

オープン
#301 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Java
スター
95
フォーク
61
平均マージ
11時間 52分
マージ済み PR(30日)
7

説明

Hello

We are configuring MSK Connect and attempting to filter data based on the EventId using the MSK Aiven connector. The setup works when pushing data from a single topic to a single index. However, it does not work when we are trying to push data from a single topic to 2 OpenSearch indices. We have tried using both the DataStream and RegexRoute options of the Aiven transformation, but have not had success.

Our goal is to load data containing EventId 4688 into the first index, and data without EventId 4688 (including all other EventIds) into the second index.

However, the connector is creating and in OpenSearch, we are not able to see a single index from this connector.

I am attaching the connector configuration for reference.

**Configuration**

transforms=Filter4688,RouteByEventId4688,FilterOthers,RouteByEventIdOthers
transforms.Filter4688.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter4688.field.value.matches=True
transforms.Filter4688.field.name=EventId
transforms.Filter4688.field.value.pattern=^4688$

transforms.RouteByEventId4688.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventId4688.regex=.*
transforms.RouteByEventId4688.replacement=topic-json-poc-4688

transforms.FilterOthers.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.FilterOthers.field.value.matches=False
transforms.FilterOthers.field.name=EventId
transforms.FilterOthers.field.value.pattern=^4688$

transforms.RouteByEventIdOthers.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventIdOthers.regex=.*
transforms.RouteByEventIdOthers.replacement=topic-json-poc-other

If anyone can help please share your input with us.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。