elastic / elastic/logstash

Add an iteration mechanism to Logstash configuration DSL (elastic/logstash)

Open
#19,095 0 comments 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

**Description**

Logstash configuration files are declarative and lack control‑flow constructs. When processing multi‑tenant data, operators often need to create multiple nearly identical plugin blocks (inputs, filters, outputs) that differ only by one parameter (e.g. a client name). For example, ingesting HR data for four clients requires four separate `file` input definitions with identical settings except for the `path` and a metadata field. As the number of clients grows, the configuration becomes long and error‑prone.

**Use Case**

Our pipeline processes JSON files located under `/datos/logs/clientes/{client}/hr/json/liquidaciones/*.json` for each client. Each input block sets `add_field => { "client" => "" }` so downstream filters know which dictionary to use. Today we must copy‑and‑paste nine `file {}` inputs (for different sub‑products) and multiple `translate {}` filters per client. Adding a new client means replicating dozens of lines. A loop or iteration construct would allow us to define a list of client names once and generate blocks dynamically:
```
# hypothetical syntax
for client in ["client_1", "client_2", "client_3", "client_4"] {
file {
path => "/datos/logs/clientes/${client}/hr/json/liquidaciones/*.json"
add_field => { "client" => "${client}" "product" => "hr" }
codec => json
}
# additional translate filters using ${client}
}
```
**Current Behavior**

Logstash’s DSL has no way to iterate or define variables. Each plugin block must be written explicitly. Operators sometimes work around this limitation by generating configuration files via external templates or scripts, but Logstash itself does not support iteration. The only parameterization available is environment or secret variable substitution and string interpolation within plugin parameters.

**Expected Behavior**

Provide a mechanism in the Logstash configuration language to define lists or maps and iterate over them to produce multiple instances of a plugin block. This mechanism should allow interpolation of the loop variable within plugin parameters. The resulting configuration should be equivalent to manually duplicating the blocks, preserving isolation between instances.

**Proposed Solution**

Consider extending the Logstash configuration DSL with a lightweight macro or looping facility. Options include:

For‑loop syntax as shown above, evaluated at pipeline compile time, expanding into multiple plugin blocks.
Template and include support, where a block can be defined once and invoked with parameters (e.g. `include "file_input_template", client => "client_1"`).
Array expansion for plugin parameters (e.g. allow an array of paths and per‑path metadata fields in the `file` input plugin). While arrays exist for some settings, they cannot assign distinct `add_field` values per element.

Any solution should maintain backward compatibility, validate loop variables, and offer clear error messages when expansion fails. By allowing iteration, Logstash pipelines for multi‑client environments will be shorter, easier to maintain and less prone to mistakes.

Contributor guide

Open the contributing guide

Research direction

No implementation files, tests, or entry points are named. Start by reviewing the Logstash configuration DSL and its compile-time expansion and validation paths, then determine which proposed mechanism is appropriate. Done means a backward-compatible iteration facility can generate isolated plugin instances with loop-variable interpolation and clear errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.