elastic / elastic/integrations

[o365] Field file.size being mapped as keyword leading to mapping conflict and rules failing

Open
#21,137 4 comments 0 reactions 0 assignees View on GitHub
bug Integration:m365_defender Integration:o365 Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Hello,

On the `o365.audit` dataset the field `file.size` is now being wrongly mapped as a `keyword` field, this field is populated from one of those fields in the ingest pipeline:

```
- set:
tag: set_file_size_723563fe
field: file.size
copy_from: o365audit.FileSizeBytes
ignore_empty_value: true
- set:
tag: set_file_size_c6196fe8
field: file.size
copy_from: o365audit.FileSize
ignore_empty_value: true
override: false
```

Since there is no explicit mapping for `file.size`, it will be dynamically mapped depending on the value from the field.

Somewhere between May and June the field `o365audit.FileSizeBytes` started arriving as a keyword instead as a long, which lead to a mapping conflict in kibana on all backing indices created after that change, which can break dashboards and security rules.

On an event from May I have this:

Image

But on an event on June I have this:

Image

So it changed from a long to a keyword, since I have a backing index with `file.size` mapped as long, this leads to mapping conflicts and break security rules, one example of a rule breaking is the native rule: _M365 OneDrive/SharePoint Excessive File Downloads_

This can be fixed with a `convert` processor to convert the `file.size` or using a explicit mapping for it.

Contributor guide

Open the contributing guide

Research direction

Start with the o365.audit ingest pipeline and inspect how o365audit.FileSizeBytes and o365audit.FileSize populate file.size, then review the resulting index mapping. Done means file.size is consistently mapped as a numeric field for new backing indices without breaking the affected dashboards or security rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.