[RFC] Basic Conditional Logic in Processors

Data Prepper currently does not provide any control logic within pipelines. This proposal is to add a basic conditional system for processing data in Preppers only when certain conditions are met.

This proposal is to add a new when field which is available for all Processors.

When a Prepper contains the when field, the condition expressed in the value must be true for the Prepper to process a given event. If the field is not provided, then all events are processed.

The conditional expression should support:

• Equality operators: ==, >, <, >=, <=
• Boolean operators: and, or, and not
• Set operators: in, and not in
• Regex operators: =~ and !~ which check a pattern on the right against the string on the left
• Fields will be accessed using JsonPointer as defined in #450
• Sets defined by [] and comma delimited

Thus, an example might be:

preppers:
  ...
  - grok:
       when: "/http/response/status_code and /http/response/status_code >= 400 and /http/response/status_code < 500"
       match: "..."
  - grok:
       when: "/http/response/status_code in [500, 501]"
       match: "..."

Implementation

The AbstractPrepper class can support the when behavior so that individual Prepper implementations do not need to handle the when field.

This will require that AbstractPrepper receive Event types and not any type. This is ongoing work in #319. Making this change in AbstractPrepper is a breaking change though since it does not require the Event type currently.

The AbstractPrepper will only call doExecute for records which meet the conditional expression provided by when.

Tasks

• Create ANTLR Parser
• Create statement evaluator
• Add when property to Abstract Processor (scope pending)
• Create logstash config converter
• Finalize scope for 1.3