elastic / elastic/apm-server

Make ingestion more resilient by ignoring malformed and avoid field explosion

Open
#11,531 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

We want to reduce the risk of mapping conflicts. The APM solution should adopt ES features to make ingestion more resilient:

(1) Add `ignore_malformed: true` to the root of the mappings. See also https://github.com/elastic/elasticsearch/pull/95329. Ignores fields with type conflicts instead of rejecting the whole document.
Validation Criteria: Ingesting a label foo: 42, then foo: "bar" doesn't lead to data loss

(2) Add `dynamic: until_limit` to avoid that field explosions lead to data loss. Relates to https://github.com/elastic/elasticsearch/pull/96235.
Validation Criteria: Exceeding the field limit by creating many unique labels doesn't lead to data loss

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.