opensearch-project / opensearch-project/data-prepper

[BUG] Ingesting example doesn't work, crashes with ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink

Open
#1,749 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
When you are attempting to docker run by guide or docker-compose-up it is crashed with
data-prepper | 2022-09-10T23:18:28,864 [main] ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink

To Reproduce
Steps to reproduce the behavior:

  1. Clone repo

  2. GOTO examples

  3. GOTO log-ingestion

  4. Follow instructions

  5. By instruction enter the command docker run --name data-prepper -v /full/path/to/log_pipeline.yaml:/usr/share/data-prepper/pipelines.yaml --network "data-prepper_opensearch-net" opensearchproject/data-prepper:latest with corresponding replacement (in my case the network name was d61fd085926e log-ingestion_opensearch-net bridge local )
    image

  6. Data-Prepper crashes with exception data-prepper | 2022-09-10T23:18:28,864 [main] ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink

(See full log attached)
CMD.report.txt

Expected behavior
Data Prepper is running correctly, you should see something similar to the following line as the latest output in your terminal.

INFO  com.amazon.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer

Screenshots
image

Environment (please complete the following information):

  • OS: Windows 11
  • Docker Desktop 4.12.0
  • VSCode

Additional context
I also did docker compose to reproduce the issue, behavior is the same with command line used and full path specified
docker-compose.txt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in examples/log-ingestion and reproduce the failure using the documented Docker command or docker-compose setup, then inspect CMD.report.txt and the attached compose configuration. Done means the OpenSearchSink no longer crashes and the container reaches the expected “No records received from buffer” output.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, java
Domain
data-engineering, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.