opensearch-project / opensearch-project/data-prepper

[BUG] bootstrap_servers is not configurable for Kafka Source in AWS

Open
#4,081 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
Creating an opensearch pipeline in AWS is not possible using the following yaml:

version: '2' kafka-pipeline: source: kafka: bootstrap_servers: - '100.100.100.100:9092' topics: - name: Topic1 group_id: groupID1 sink: - opensearch: hosts: [ "https://XXXX.eu-central-1.amazonaws.com" ] aws: sts_role_arn: "arn:aws:iam::XXXXXXX:role/AWSROLE" region: "eu-central-1" index: "index_123"

It returns an error with the following message:

"kafka.bootstrap_servers" is not configurable parameter for Amazon OpenSearch Ingestion pipelines using an aws kafka source: "$['log-pipeline']['source']['kafka']".

To Reproduce
Steps to reproduce the behavior:

  1. Go to Opensearch Ingestion Pipelines
  2. Create a new pipeline
  3. Paste the yaml and validate
  4. See error

Expected behavior
It should be valid and start a pipeline

Screenshots
SCR-20240203-pozj

Environment (please complete the following information):

  • AWS eu-central-1
  • OpenSearch Cluster Version 2.11 (AWS Latest)
  • Service Software Version: OpenSearch_2_11_R20231113-P2 (latest)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Kafka source configuration and the AWS OpenSearch Ingestion validation that rejects bootstrap_servers. Reproduce the reported YAML validation error, then verify that the setting is accepted and the pipeline can start without the configurable-parameter error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kafka
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.