Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Unable to push data from one MSK topic into two OpenSearch indices

Offen
#301 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
95
Forks
61
Ø Merge
11 Std. 52 Min.
Gemergte PRs (30 T.)
7

Beschreibung

Hello

We are configuring MSK Connect and attempting to filter data based on the EventId using the MSK Aiven connector. The setup works when pushing data from a single topic to a single index. However, it does not work when we are trying to push data from a single topic to 2 OpenSearch indices. We have tried using both the DataStream and RegexRoute options of the Aiven transformation, but have not had success.

Our goal is to load data containing EventId 4688 into the first index, and data without EventId 4688 (including all other EventIds) into the second index.

However, the connector is creating and in OpenSearch, we are not able to see a single index from this connector.

I am attaching the connector configuration for reference.

**Configuration**

transforms=Filter4688,RouteByEventId4688,FilterOthers,RouteByEventIdOthers
transforms.Filter4688.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter4688.field.value.matches=True
transforms.Filter4688.field.name=EventId
transforms.Filter4688.field.value.pattern=^4688$

transforms.RouteByEventId4688.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventId4688.regex=.*
transforms.RouteByEventId4688.replacement=topic-json-poc-4688

transforms.FilterOthers.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.FilterOthers.field.value.matches=False
transforms.FilterOthers.field.name=EventId
transforms.FilterOthers.field.value.pattern=^4688$

transforms.RouteByEventIdOthers.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventIdOthers.regex=.*
transforms.RouteByEventIdOthers.replacement=topic-json-poc-other

If anyone can help please share your input with us.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.