Add an option to populate event.created on all inputs.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
The description of the field in the ECS documentation is
```
event.created contains the date/time when the event was first read by an agent, or by your pipeline.
This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event.
```
From the couple of inputs I've checked there isn't an option to do this. The next best option is adding it within the first filter of the pipeline. However this isn't exactly what is intended for the field (though a few ms difference is trivial) and could, at least conceptually, obfuscate any delays between the input and the next filter adding the field.
What I am suggesting is adding an option to each input plugin that will populate `[event][created]` with the current time at ingestion. This aligns with the intended use of the field and improves its use for pipeline monitoring.
This probably should be an optional field that defaults to false to avoid issues for pipelines that have already added the field or do not want to use it. Possibly, the field name itself could also differ (or be a setting) depending on the input plugin/pipeline's ecs_compatibility settings.
There may also need to consideration of handling input that may already have it included (beats, agent, another logstash node) and whether to skip setting it or overwrite it. In terms of ecs it's probably better to skip that unless the pipeline's specific use treats this ingestion as the creation. Though, largely, I am guessing any of those cases just would not set the option if it was already set in the pipeline.
Contributor guide
Research direction
Start with the ECS documentation definition of event.created and compare the input plugins with the first-filter pipeline approach described here. Resolve the option's default, ECS field naming, and behavior when event.created already exists; the issue is done when the input behavior and coverage across inputs are agreed and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100