Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Unable to push data from one MSK topic into two OpenSearch indices

Ouverte
#301 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
95
Forks
61
Merge moyen
11 h 52 min
PR mergées (30 j)
7

Description

Hello

We are configuring MSK Connect and attempting to filter data based on the EventId using the MSK Aiven connector. The setup works when pushing data from a single topic to a single index. However, it does not work when we are trying to push data from a single topic to 2 OpenSearch indices. We have tried using both the DataStream and RegexRoute options of the Aiven transformation, but have not had success.

Our goal is to load data containing EventId 4688 into the first index, and data without EventId 4688 (including all other EventIds) into the second index.

However, the connector is creating and in OpenSearch, we are not able to see a single index from this connector.

I am attaching the connector configuration for reference.

**Configuration**

transforms=Filter4688,RouteByEventId4688,FilterOthers,RouteByEventIdOthers
transforms.Filter4688.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter4688.field.value.matches=True
transforms.Filter4688.field.name=EventId
transforms.Filter4688.field.value.pattern=^4688$

transforms.RouteByEventId4688.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventId4688.regex=.*
transforms.RouteByEventId4688.replacement=topic-json-poc-4688

transforms.FilterOthers.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.FilterOthers.field.value.matches=False
transforms.FilterOthers.field.name=EventId
transforms.FilterOthers.field.value.pattern=^4688$

transforms.RouteByEventIdOthers.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventIdOthers.regex=.*
transforms.RouteByEventIdOthers.replacement=topic-json-poc-other

If anyone can help please share your input with us.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.