Pipeline Component Ids

Introduction

A Data Prepper pipeline can contain multiple sources, processors, and sinks with the same type. Presently, these cannot be distinguished.

Proposed Solution

Data Prepper should assign a unique identifier for each pipeline component. The scope of the Id is within the current pipeline. There will be a fully-qualified Id, which is discussed at the end of this issue description. For most of this discussion, the Id is unique only within a single pipeline.

Additionally, pipeline authors may wish to configure some component Ids. This can help them debug their pipelines and make them more readable.

The following example shows how a pipeline author can configure the Id using an id property:

log-pipeline:
  source:
    http:
  prepper:
    - grok:
        id: extract-apache-logs
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        id: opensearch-a
        hosts: [ "https://opensearch-host-a" ]
    - opensearch:
        id: opensearch-a
        hosts: [ "https://opensearch-host-b" ]

Pipeline authors do not need to configure the id. Data Prepper will produce a default value.

Id Generation

The default Id generation should be deterministic. This will allow the peer-forwarder to use the id of a component and consistently supply Events to the correct component in a peer node.

The default Id generation can be:

${pluginType}${incrementedCount > 1 ? incrementedCount : ''}

The incrementedCount will be a number which is incremented for each component type individually. It can thus be stored in a map: Map<String, Integer> typeToIncrementedCount. The count will be incremented before applying the function above. So the first of any given type has incrementedCount == 1. This approach allows the pipelines without duplicates to continue to use the pluginType without having a trailing 1.

Examples

No Configured Ids

log-pipeline:
  source:
    http:
  prepper:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch-host-a" ]
    - opensearch:
        hosts: [ "https://opensearch-host-b" ]

The Ids are:

• http
• grok
• opensearch
• opensearch2

Some Configured Ids

log-pipeline:
  source:
    http:
  prepper:
    - grok:
        match:
          id: extract-apache-logs
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        id: opensearch-a
        hosts: [ "https://opensearch-host-a" ]
    - opensearch:
        hosts: [ "https://opensearch-host-b" ]

The Ids are:

• http
• extract-apache-logs
• opensearch-a
• opensearch2

Alternatives

Duplicates Always Have Count Suffix

Another approach is to identify any plugin type that has more than one plugin in the pipeline. Only those that have more than one will have a suffix. This can be nice because each plugin of the same type has a more consistent name.

The disadvantage is that it may be more complicated to support. Is the improvement to the name really worthwhile here? Pipeline authors who want better names can control the id already.

log-pipeline:
  source:
    http:
  prepper:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch-host-a" ]
    - opensearch:
        hosts: [ "https://opensearch-host-b" ]

The Ids are:

• http
• grok
• opensearch1
• opensearch2

Count Across Components

Data Prepper could increment a universal count. The disadvantage is that when there is only one processor of a type it gets some number behind it.

log-pipeline:
  source:
    http:
  prepper:
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch-host-a" ]
    - opensearch:
        hosts: [ "https://opensearch-host-b" ]

The Ids are:

• http1
• grok2
• opensearch3
• opensearch4

Fully Scoped Ids

Data Prepper will generate and validate plugin Ids only within a single plugin. Additionally, Data Prepper will support fully qualified component Ids. A fully-qualified plugin Id will be unique across all pipelines. The format will be:

{pipelineName}.{pluginId}

This format is based on the current convention for plugin metrics. Data Prepper currently defines metrics by:

{pipelineName}.{pluginType}.{metricName}

Tasks

• #3995
• Allow users to set component Ids
• Update metrics to include update component Ids