Make ingestion more resilient by ignoring malformed and avoid field explosion
Open
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
We want to reduce the risk of mapping conflicts. The APM solution should adopt ES features to make ingestion more resilient:
(1) Add `ignore_malformed: true` to the root of the mappings. See also https://github.com/elastic/elasticsearch/pull/95329. Ignores fields with type conflicts instead of rejecting the whole document.
Validation Criteria: Ingesting a label foo: 42, then foo: "bar" doesn't lead to data loss
(2) Add `dynamic: until_limit` to avoid that field explosions lead to data loss. Relates to https://github.com/elastic/elasticsearch/pull/96235.
Validation Criteria: Exceeding the field limit by creating many unique labels doesn't lead to data loss
Contributor guide
Assessment
This issue has not been assessed yet.