opensearch-project / opensearch-project/data-prepper

Add Data Prepper internal tags

Open
#3,956 0 comments 0 reactions 1 assignee View on GitHub

@kkondaka is already working on this.

Since Jan 11, 2024.

Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
Feature request is to add support for data prepper internal tags which can be used for different purposes.

Describe the solution you'd like
This feature would allow adding Data Prepper internal tags, identified by having a fixed data prepper internal prefix (like _data_prepper_internal_). These tags may be added by any part of the Data Prepper (api, core, sources, sinks, processor, etc) and these tags may be used either internally or by pipeline config various use case 1.

  1. Routing logic in Data Prepper core could tag routed events, and pipeline code could check for events that were not routed (with tag like _data_prepper_internal_routed and take action on those events (like dropping them or sending them to DLQ)
  2. Failures at any part of the pipeline could add an internal tag (like _data_prepper_internal_to_dlq) to send the failed events to pipeline DLQ.

To prevent sending these tags to the Sink, all the tags with _data_prepper_internal_ prefix can be filtered out

Describe alternatives you've considered (Optional)
Alternate solution would be to introduce a new field in EventMetadata like internalTags (need not even be called tags, it could be flags or something like that), which can be used for the same purpose. But it would require all the expressions, conditions support that event tags already have. I think this would result in a lot of duplicate/similar code and double the effort for any functionality we add for tags.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.