elastic / elastic/integrations

Should auditd.messages field be mapped as text or match_only_text?

Open
#20,836 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Currently the `auditd.messages` [field](https://github.com/elastic/integrations/blob/main/packages/auditd_manager/data_stream/auditd/fields/fields.yml#L69) is mapped as keyword field.

We have seen a merge of a single `auditd.messages` field take up to 400+ mb of jvm heap. Estimated unique number of messages was ~260M. This made a node crash. The very high number of unique values is likely a consequence because the `auditd.messages` is mapped as a keyword field. If it was mapped as match_only_text or text field type, this wouldn't have happened.

If `auditd.messages` is a real message field than a text like field type matches better as well, because each message field is likely unique (b/c of timestamp)? A text like field type would better fits the needs for that field. Keyword field type would only allow for exact matching.

Contributor guide

Open the contributing guide

Research direction

Start with packages/auditd_manager/data_stream/auditd/fields/fields.yml at the auditd.messages field around line 69, then review the reported heap impact and the differences between keyword, text, and match_only_text mappings. Done means selecting and applying a mapping that fits message-field search needs without reproducing the reported high-cardinality heap problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.