elastic / elastic/integrations
[Suricata]: field suricata.eve.stats.flow.end is incorrecly assumed as date
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Suricata [suricata]
### Dataset Name
suricata.eve
### Integration Version
2.27.0
### Agent Version
8.19.6
### Agent Output Type
logstash
### Elasticsearch Version
8.19.6
### OS Version and Architecture
Debian 12.12 (x86_64)
### Software/API Version
Suricata 7.0.8
### Error Message
Error message was: Elastichsearch was expecting the field suricata.eve.stats.flow.end to be of type 'date', but instead it was 'object'
### Event Original
No actual log available, but it's the suricata json-format
### What did you do?
Integration configured with defaults, only path of the logfile was changed. In suricata, the 'stats' is enabled.
### What did you see?
Events from logstash to elasticsearch were dropped into dead letter queue if they included the "stats" -block.
### What did you expect to see?
Events in the datastream
### Anything else?
After some debugging, there seems to be faulty mapping in the component template `logs-suricata.eve@package`. The problem is this block:
```
{
"_embedded_ecs-end_to_date": {
"mapping": {
"type": "date"
},
"match": "end"
}
},
```
But actually it probably is inherited from `ecs@mappings` template's `ecs_date` -block;
```
"ecs_date": {
"path_match": [
"*.timestamp",
"*_timestamp",
"*.not_after",
"*.not_before",
"*.accessed",
"created",
"*.created",
"*.installed",
"*.creation_date",
"*.ctime",
"*.mtime",
"ingested",
"*.ingested",
"*.start",
"*.end",
"*.indicator.first_seen",
"*.indicator.last_seen",
"*.indicator.modified_at",
"*threat.enrichments.matched.occurred"
],
"mapping": {
"type": "date"
},
"unmatch_mapping_type": "object"
}
```
Which matches (?) all fields named 'end' and tries to map those as type `date`. But the field `suricata.eve.stats.flow.end` is not in fact a date, but an object, as per: https://docs.suricata.io/en/suricata-8.0.2/appendix/eve-index.html#stats-flow-end-object
To summarize: all fields named 'end' are not timestamps and especially not in Suricata -context.
Contributor guide
Assessment
This issue has not been assessed yet.