opensearch-project / opensearch-project/data-prepper
Log ingestion example does not work
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Not sure where to ask this as it seems the engagement is low on forums like StackOverflow.
I am looking to make the move from ELK to OpenSearch. The purpose is solely for application log aggregation. I can not find a simple example of how to do this however...the example discussed here https://opensearch.org/docs/latest/observing-your-data/log-ingestion/, does not work and it seems like that documentation is out of date based on what the docker-compose.yml it refers to looks like. In particular the repo does not show any 'data-prepper' in the docker-compose.yml. Additionally, while the example does build and start 3 containers- it does not seem to push any data to opensearch when you make modifications to the test.log file.
So, here i am..asking really just for something that actually works. anything out there? I have been warned that the documentation for this project is very lacking - but i feel like there should be one basic example that new users could follow to get working...just one.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the log-ingestion documentation linked in the issue and compare it with the referenced docker-compose.yml, noting the missing data-prepper service. Run the example and modify test.log to trace whether data reaches OpenSearch. Done means the documented example builds, starts, and visibly ingests the test log data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose
- Domain
- devops, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100