opensearch-project / opensearch-project/data-prepper

[BUG] Processing A Nested List as Individual Log Events

Open
#5,015 11 comments 2 reactions 1 assignee View on GitHub

@oeyh is already working on this.

Since Oct 8, 2024.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
It could potentially possible to do this however I have not been able to find anything in documentation that covers it.

If you have CloudWatch Logs -> Data Firehose -> S3 and want to pull that into DataPrepper it brings in the multi line event.

The structure seems to be like so:

{ "messageType": "DATA_MESSAGE", "owner": "123456789", "logGroup": "foo", "logStream": "bar", "logEvents": [{"id": "123456", "message": "some log message here", "timestamp" 1727880215114}, {"id": "789102", "message": "another log message here", "timestamp" 1727880215114}, {"id": "99999", "message": "yet another log message here", "timestamp" 1727880215114}]}

What I was hoping to do was use DataPrepper to read in the log message from S3 (that is like above) and then parse out the "logEvents" and treat each entry as an individual log message to publish to S3 & OpenSearch alike.

S3 being it will allow me to create a neat structure of a prefix with accountid/log-group/YYYY/MM/DD/HH

However I am not sure that it is possible to extract logEvents dictionary that contains a list of arrays and treat them as separate events.

To Reproduce
Steps to reproduce the behavior:

  1. Create fake log event like above.
  2. Write to DataPrepper
  3. Try to parse

Expected behavior
I was expecting a feature within DataPrepper to support something like so:

  processor:
    - parse_json:
    - split_string:
        entries:
          - source: "/logEvents[0]"
            delimiter: ","

Environment (please complete the following information):

  • OS: macOSX

Additional context
AWS Managed OpenSearch & AWS Managed OSIS is being used. I setup a local container deployment to expedite testing and still see the same issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.