opensearch-project / opensearch-project/data-prepper

Failed Initialization of Opensearch Sink

Open
#3,810 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

I am made two files, docker-compose.yml file with following description
version: '3'
services:
data-prepper:
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./log_pipeline.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
ports:
- 2021:2021
networks:
- opensearch-net
networks:
opensearch-net:

and log_pipelines.yaml with following description
log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ "%{COMMONAPACHELOG}" ]
sink:
- opensearch:
hosts: [ "https://opensearch.stack.com:9200" ]
insecure: true
username: admin
password: admin
index: nginx_logs

upon running docker compose file, the container is running but when i check it logs, having following problem

2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-06T07:13:10,587 [Thread-2] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: opensearch.stack.com
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-06T07:13:11,587 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: opensearch.stack.com

i am using latest version for pulling image, is this causing an error

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docker-compose.yml and log_pipelines.yaml, then review the Data Prepper container logs shown in the issue. Verify the OpenSearch sink configuration and container connectivity using the logged initialization retry as the starting symptom. Done means the sink initializes successfully and the pipeline no longer reports that it is not ready.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose
Domain
devops, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.