opensearch-project / opensearch-project/data-prepper

Support live tracking of event transformation

Open
#4,711 0 comments 0 reactions 1 assignee View on GitHub

@kkondaka is already working on this.

Since Jul 8, 2024.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
Currently, DataPrepper users do not have a way to see how an event is created and transformed in the pipeline until the event is written to a sink.
It would help users to see how an event gets created and transformed by different components in the pipeline

Describe the solution you'd like
Solution is to allow capturing of a sampled event transformations at various places in the DataPrepper and send it a user configured endpoint (any sink that is currently supported).
The final live capture event would be a collection of events that are captured at the various stages of the pipeline, like, at the time of event creation, after every processor, after routing decisions, after codecs, and so on.
The live capturing event state can be disabled by default and be enabled by a control command. When enabled, one of the events received is "marked" for tracking periodically (for example, 1 event per second) and the marked event's metadata is populated with the live capture information. When the event is finally, released, the captured data is sent to the configured endpoint. If an event gets copied as part of routing (and sub-pipelines), then the entire live capture information accumulated so far is also copied to the new event. The final live capture event may look like this

{"LiveCapture":
   [
    
    {
        "Version": "1.0",
        "Time": Time in ISO-8601 format,
        "description": "Source XYZ created event",
        "event": {
             "key1" : "value1",
             "key2" : "value2"
        }
    },
    {
        {
        "Version": "1.0",
        "Time": Time in ISO-8601 format,
        "description": " after add entries processor execution",
        "event": {
             "key1" : "value1",
             "key2" : "value2",
             "key3": "value3"
        }
    },
    {
        {
        "Version": "1.0",
        "Time": Time in ISO-8601 format,
        "description": " Event matched the route <route-name>",
        "event": {
             "key1" : "value1",
             "key2" : "value2",
             "key3": "value3"
        }
    },
    {
        "Version": "1.0",
        "Time": Time in ISO-8601 format,
        "description": " Received by sink <sink-name>",
        "event": {
             "key1" : "value1",
             "key2" : "value2",
             "key3": "value3"
        }
    },
    {
        "Version": "1.0",
        "Time": Time in ISO-8601 format,
        "description": " sink <sink-name> modified the event with include_tags",
        "event": {
             "key1" : "value1",
             "key2" : "value2",
             "key3" : "value3",
             "tags" : [tag1, tag2]
        }
    }
   ]
}


Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.