elastic / elastic/apm

Add ability to capture non-indexed large content for spans

Open
#752 6 comments 7 reactions 0 assignees View on GitHub
discuss discussion
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

We currently have a distinction between large custom context data that is not intended for indexing and such that is purposed for indexing. The former can be added through [agents API](https://www.elastic.co/guide/en/apm/guide/current/data-model-metadata.html#data-model-custom), but only to transactions and errors. The latter, [labels](https://www.elastic.co/guide/en/apm/guide/current/data-model-metadata.html#data-model-labels), are applicable for spans as well, but are limited in size.
There are cases where users want to add large textual data to spans' context, without getting it indexed, for example- as described in this [forum request](https://discuss.elastic.co/t/span-level-custom-context-metadata/323868).

**Describe the solution you'd like**

We should consider adding the [custom context](https://www.elastic.co/guide/en/apm/guide/current/data-model-metadata.html#data-model-custom) API to spans as well. It will have the same limitations as the existing one has in terms of maximum allowed size and not being indexed.
In order to offer this capability through OTel API as well, we may decide on a specific attribute that will be mapped to the custom context field that has the proper ES mapping.

For now, this issue is only for 👍 and 👎 or any further discussion. A spec PR will follow if we decide that it is useful and important enough.

**Describe alternatives you've considered**

Another alternative is to use the labels API for custom context data as well. However, this requires a change to the current [ECS definitions for the mapping of this field](https://github.com/elastic/ecs/blob/1794cfb26f4f3942207ba05a854cca77bf814dd2/generated/beats/fields.ecs.yml#L24-L35), adding the `ignore_above: 1024` to it. This will allow automatic mapping decision based on size. However, since sending very large labels to older APM indices may be problematic, it will require agents to be aware of the actual backend mapping, which would probably be an overkill. If it was only dependent on the APM Server version, that would be OK, as agents are already aware of that (required for the numeric and boolean labels), but this means also a change in APM Server to override the ECS mapping for this field.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.