elastic / elastic/integrations
Should auditd.messages field be mapped as text or match_only_text?
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
Currently the `auditd.messages` [field](https://github.com/elastic/integrations/blob/main/packages/auditd_manager/data_stream/auditd/fields/fields.yml#L69) is mapped as keyword field.
We have seen a merge of a single `auditd.messages` field take up to 400+ mb of jvm heap. Estimated unique number of messages was ~260M. This made a node crash. The very high number of unique values is likely a consequence because the `auditd.messages` is mapped as a keyword field. If it was mapped as match_only_text or text field type, this wouldn't have happened.
If `auditd.messages` is a real message field than a text like field type matches better as well, because each message field is likely unique (b/c of timestamp)? A text like field type would better fits the needs for that field. Keyword field type would only allow for exact matching.
Contributor guide
Research direction
Start with packages/auditd_manager/data_stream/auditd/fields/fields.yml at the auditd.messages field around line 69, then review the reported heap impact and the differences between keyword, text, and match_only_text mappings. Done means selecting and applying a mapping that fits message-field search needs without reproducing the reported high-cardinality heap problem.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100