opensearch-project / opensearch-project/data-prepper
[BUG] bootstrap_servers is not configurable for Kafka Source in AWS
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
Creating an opensearch pipeline in AWS is not possible using the following yaml:
version: '2'
kafka-pipeline:
source:
kafka:
bootstrap_servers:
- '100.100.100.100:9092'
topics:
- name: Topic1
group_id: groupID1
sink:
- opensearch:
hosts: [ "https://XXXX.eu-central-1.amazonaws.com" ]
aws:
sts_role_arn: "arn:aws:iam::XXXXXXX:role/AWSROLE"
region: "eu-central-1"
index: "index_123"
It returns an error with the following message:
"kafka.bootstrap_servers" is not configurable parameter for Amazon OpenSearch Ingestion pipelines using an aws kafka source: "$['log-pipeline']['source']['kafka']".
To Reproduce
Steps to reproduce the behavior:
- Go to Opensearch Ingestion Pipelines
- Create a new pipeline
- Paste the yaml and validate
- See error
Expected behavior
It should be valid and start a pipeline
Screenshots
Environment (please complete the following information):
- AWS eu-central-1
- OpenSearch Cluster Version 2.11 (AWS Latest)
- Service Software Version: OpenSearch_2_11_R20231113-P2 (latest)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Kafka source configuration and the AWS OpenSearch Ingestion validation that rejects bootstrap_servers. Reproduce the reported YAML validation error, then verify that the setting is accepted and the pipeline can start without the configurable-parameter error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kafka
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100