opensearch-project / opensearch-project/data-prepper
[BUG] Processing A Nested List as Individual Log Events
@oeyh is already working on this.
Since Oct 8, 2024.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
It could potentially possible to do this however I have not been able to find anything in documentation that covers it.
If you have CloudWatch Logs -> Data Firehose -> S3 and want to pull that into DataPrepper it brings in the multi line event.
The structure seems to be like so:
{ "messageType": "DATA_MESSAGE", "owner": "123456789", "logGroup": "foo", "logStream": "bar", "logEvents": [{"id": "123456", "message": "some log message here", "timestamp" 1727880215114}, {"id": "789102", "message": "another log message here", "timestamp" 1727880215114}, {"id": "99999", "message": "yet another log message here", "timestamp" 1727880215114}]}
What I was hoping to do was use DataPrepper to read in the log message from S3 (that is like above) and then parse out the "logEvents" and treat each entry as an individual log message to publish to S3 & OpenSearch alike.
S3 being it will allow me to create a neat structure of a prefix with accountid/log-group/YYYY/MM/DD/HH
However I am not sure that it is possible to extract logEvents dictionary that contains a list of arrays and treat them as separate events.
To Reproduce
Steps to reproduce the behavior:
- Create fake log event like above.
- Write to DataPrepper
- Try to parse
Expected behavior
I was expecting a feature within DataPrepper to support something like so:
processor:
- parse_json:
- split_string:
entries:
- source: "/logEvents[0]"
delimiter: ","
Environment (please complete the following information):
- OS: macOSX
Additional context
AWS Managed OpenSearch & AWS Managed OSIS is being used. I setup a local container deployment to expedite testing and still see the same issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.