elastic / elastic/integrations
[o365] Field file.size being mapped as keyword leading to mapping conflict and rules failing
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
Hello,
On the `o365.audit` dataset the field `file.size` is now being wrongly mapped as a `keyword` field, this field is populated from one of those fields in the ingest pipeline:
```
- set:
tag: set_file_size_723563fe
field: file.size
copy_from: o365audit.FileSizeBytes
ignore_empty_value: true
- set:
tag: set_file_size_c6196fe8
field: file.size
copy_from: o365audit.FileSize
ignore_empty_value: true
override: false
```
Since there is no explicit mapping for `file.size`, it will be dynamically mapped depending on the value from the field.
Somewhere between May and June the field `o365audit.FileSizeBytes` started arriving as a keyword instead as a long, which lead to a mapping conflict in kibana on all backing indices created after that change, which can break dashboards and security rules.
On an event from May I have this:
But on an event on June I have this:
So it changed from a long to a keyword, since I have a backing index with `file.size` mapped as long, this leads to mapping conflicts and break security rules, one example of a rule breaking is the native rule: _M365 OneDrive/SharePoint Excessive File Downloads_
This can be fixed with a `convert` processor to convert the `file.size` or using a explicit mapping for it.
Contributor guide
Research direction
Start with the o365.audit ingest pipeline and inspect how o365audit.FileSizeBytes and o365audit.FileSize populate file.size, then review the resulting index mapping. Done means file.size is consistently mapped as a numeric field for new backing indices without breaking the affected dashboards or security rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100