elastic / elastic/cloudbeat

JSON stringify unmapped fields

Open
#2,459 0 comments 1 reaction 0 assignees View on GitHub
cloudbeat enhancement Team:Cloud Security
Dominant language
Go
Stars
58
Forks
55
Avg merge
9h 9m
Merged PRs (30d)
424

Description

**Motivation**

Today we don't map `resource.raw`, `result.evidence` and `result.expected` because we these are sent as part of the source, if we will apply dynamic mappings on these fields we reach to the default max limit of number of fields being mapped and we experienced mapping collision which drops the document from being indexed.

So our solution at this time was not to map them, so we at least be able to present these values to the user because they are part of _source.

However, there could be a better solution for this problem which is to map these fields as text and their value will be a stringified JSON

1. It keeps the same behavior from user perspective as it is today
2. This is our only use of _source in kibana, which blocks us from using synthetic source which optimizes the storage of these indices
3. It will allow the user (and us) to query these fields as text which is an improvement
4. In the future, es|ql might support querying these fields with `jq` kind of query language ([reference](https://github.com/elastic/elasticsearch/issues/104934))

**Definition of done**

- [ ] Add mapping as text to cloud_security_package to the fields mentioned above
- [ ] Cloudbeat should send this values as a valid stringified JSON

**Out of scope**

-

**Related tasks/epics**

- [LogsDB Index Mode Release Plan - DRAFT](https://docs.google.com/document/d/1wDkYv37ExvaN3Qm2Z7G5bWjtVIFj7Be2s5ZIZxlRhcw/edit#heading=h.tp580vllaopp)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.