opensearch-project / opensearch-project/data-prepper
Failed Initialization of Opensearch Sink
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
I am made two files, docker-compose.yml file with following description
version: '3'
services:
data-prepper:
container_name: data-prepper
image: opensearchproject/data-prepper:latest
volumes:
- ./log_pipeline.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
ports:
- 2021:2021
networks:
- opensearch-net
networks:
opensearch-net:
and log_pipelines.yaml with following description
log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ "%{COMMONAPACHELOG}" ]
sink:
- opensearch:
hosts: [ "https://opensearch.stack.com:9200" ]
insecure: true
username: admin
password: admin
index: nginx_logs
upon running docker compose file, the container is running but when i check it logs, having following problem
2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-06T07:13:10,583 [Thread-2] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-06T07:13:10,587 [Thread-2] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: opensearch.stack.com
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - sink is not ready for execution, retrying
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2023-12-06T07:13:11,582 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2023-12-06T07:13:11,587 [log-pipeline-sink-worker-2-thread-1] WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: opensearch.stack.com
i am using latest version for pulling image, is this causing an error
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docker-compose.yml and log_pipelines.yaml, then review the Data Prepper container logs shown in the issue. Verify the OpenSearch sink configuration and container connectivity using the logged initialization retry as the starting symptom. Done means the sink initializes successfully and the pipeline no longer reports that it is not ready.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose
- Domain
- devops, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100