opensearch-project / opensearch-project/data-prepper

Log ingestion example does not work

Open
#2,640 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Not sure where to ask this as it seems the engagement is low on forums like StackOverflow.
I am looking to make the move from ELK to OpenSearch. The purpose is solely for application log aggregation. I can not find a simple example of how to do this however...the example discussed here https://opensearch.org/docs/latest/observing-your-data/log-ingestion/, does not work and it seems like that documentation is out of date based on what the docker-compose.yml it refers to looks like. In particular the repo does not show any 'data-prepper' in the docker-compose.yml. Additionally, while the example does build and start 3 containers- it does not seem to push any data to opensearch when you make modifications to the test.log file.

So, here i am..asking really just for something that actually works. anything out there? I have been warned that the documentation for this project is very lacking - but i feel like there should be one basic example that new users could follow to get working...just one.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the log-ingestion documentation linked in the issue and compare it with the referenced docker-compose.yml, noting the missing data-prepper service. Run the example and modify test.log to trace whether data reaches OpenSearch. Done means the documented example builds, starts, and visibly ingests the test log data.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
devops, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.