elastic / elastic/logstash

Add an option to populate event.created on all inputs.

Open
#15,865 1 comment 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

The description of the field in the ECS documentation is
```

event.created contains the date/time when the event was first read by an agent, or by your pipeline.

This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.

```

From the couple of inputs I've checked there isn't an option to do this. The next best option is adding it within the first filter of the pipeline. However this isn't exactly what is intended for the field (though a few ms difference is trivial) and could, at least conceptually, obfuscate any delays between the input and the next filter adding the field.

What I am suggesting is adding an option to each input plugin that will populate `[event][created]` with the current time at ingestion. This aligns with the intended use of the field and improves its use for pipeline monitoring.

This probably should be an optional field that defaults to false to avoid issues for pipelines that have already added the field or do not want to use it. Possibly, the field name itself could also differ (or be a setting) depending on the input plugin/pipeline's ecs_compatibility settings.

There may also need to consideration of handling input that may already have it included (beats, agent, another logstash node) and whether to skip setting it or overwrite it. In terms of ecs it's probably better to skip that unless the pipeline's specific use treats this ingestion as the creation. Though, largely, I am guessing any of those cases just would not set the option if it was already set in the pipeline.

Contributor guide

Open the contributing guide

Research direction

Start with the ECS documentation definition of event.created and compare the input plugins with the first-filter pipeline approach described here. Resolve the option's default, ECS field naming, and behavior when event.created already exists; the issue is done when the input behavior and coverage across inputs are agreed and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.