Aiven-Open / Aiven-Open/transforms-for-apache-kafka-connect

Filter operation is not working.

Open
#126 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
90
Forks
29
PR merge metrics
No merged PRs in 30d

Description

We are currently setting up the MSK Connect pipeline to transfer data from a Kafka Topic to OpenSearch using the Aiven connector as a plugin. We have successfully created a topic and produced dummy data on that topic. Additionally, we've configured an MSK connector to read and write the data, successfully writing it into OpenSearch.

However, when attempting to filter the data based on a specific field, data is not being written to OpenSearch(It is also not creating the Index in OpenSearch). Below is the configuration for the MSK Connect:

connector.class=io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
type.name=kafka-connect
connection.password=***********
transforms.Filter.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter.field.name=EventId
tasks.max=1
topics=json-topic
connection.username=sample
transforms=Filter
bootstrap.servers="***********************************************"
key.ignore=true
schema.ignore=true
key.converter.schemas.enable=false
value.converter.schemas.enable=false
transforms.Filter.field.value=4690
transforms.Filter.field.value.matches=true
value.converter=org.apache.kafka.connect.json.JsonConverter
connection.url="*********************"
key.converter=org.apache.kafka.connect.storage.StringConverter

It is not working. I am not getting any errors but the filter data is not written to OpenSearch. I checked the data in topic also it contains the event id 4690.

Can you please help us.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.