Metricbeat `aws.tags.*` dynamic template incompatible with dotted AWS tag keys
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
- Version: Metricbeat 8.17 and above
### Problem
As of 8.17, Beats can access AWS instance tags with the `add_cloud_metadata` processor
https://github.com/elastic/beats/pull/41477
The AWS tags from EC2 hosts are added under `aws.tags`.
AWS does not restrict tag keys from containing dots (`.`), but Metricbeat's default dynamic template for `aws.tags` is defined as:
```
{
"aws.tags.*": {
"path_match": "aws.tags.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "*"
}
}
```
This mapping applies to all types (`match_mapping_type: "*"`) including objects.
If an AWS tag key contains a dot, Elasticsearch interprets it as a nested object field.
For example, the tag key `csg.support` and value `something` results in:
```
{
"aws": {
"tags": {
"csg": {
"support": "something"
}
}
}
}
```
Due to the MB template, Elasticsearch tries to map `aws.tags.csg` (an object) as a `keyword`, which results in parsing errors
```
{
"error": {
"root_cause": [
{
"type": "document_parsing_exception",
"reason": "[1:28] failed to parse field [aws.tags.csg] of type [keyword] in document with id '1'. Preview of field's value: '{support=something}'"
}
],
"type": "document_parsing_exception",
"reason": "[1:28] failed to parse field [aws.tags.csg] of type [keyword] in document with id '1'. Preview of field's value: '{support=something}'",
"caused_by": {
"type": "illegal_state_exception",
"reason": "Can't get text on a START_OBJECT at 1:4"
}
},
"status": 400
}
```
### Impact
Ingest will break in any environment that uses dots in their AWS tagging convention and have the `add_cloud_metadata` processor enabled. This can unexpectedly happen on an upgrade from an earlier version to 8.17 since the tags will suddenly become accessible to the processor.
Incidentally, this has been observed in ECE 3.8.0 (where 8.18 beats are used and users do not have control over the templates), and has partially broken the ingest to the system logging and metrics cluster. Which has some other downstream UI issues.
### Suggestion
I'm not sure what the best approach is TBH. but it doesn't seem like the current approach fits within ECS guidelines https://www.elastic.co/docs/reference/ecs/ecs-guidelines#_guidelines_for_field_names
It says to reserves dots to indicate object hierarchy, and requires that a field path resolve to a single consistent type. But the current template allows dotted tag keys to be mapped inconsistently as keywords or objects(though this results in indexing errors)..
If this doesn't end up meriting a change, then perhaps we can add a Special Note to the add cloud metadata documentation about this consideration: https://www.elastic.co/docs/reference/beats/filebeat/add-cloud-metadata
Some ideas...
1. Change the default dynamic template for `aws.tags` to only apply to string fields instead of all types with `match_mapping_type": "string"`. This avoids attempting to force objects into `keyword` amd keeps existing behavior for string values while preventing mapping conflicts and ingestion failures.
But that would still susceptible to path type conflicts if a tag key evolves from a leaf string to an object (e.g., first `csg: "something"`, later `csg.support: "something"`)
2. So perhaps mapping `aws.tags` as `flattened` instead could be better to tolerate arbitrary depth without explosions and conflicts. Problem with this it changes the query semantics.
---
### Steps to Reproduce:
install metricbeat on an EC2 instance with tags allowed in metadata and a tag that contains a dot in the key.
Or we we can simulate with these APIs
Create index with the mapping in metricbeat's default template
```
PUT aws-tag-test/
{
"mappings": {
"dynamic_templates": [
{
"aws.tags.*": {
"path_match": "aws.tags.*",
"mapping": {
"type": "keyword"
},
"match_mapping_type": "*"
}
}
]
}
}
```
Add a document mimicking a dotted tag key. This will error.
```
PUT aws-tag-test/_doc/1
{
"aws.tags.csg.support": "something"
}
```
Contributor guide
Research direction
The issue identifies the default aws.tags dynamic template and the add_cloud_metadata processor as starting points; reproduce the failure with the supplied aws-tag-test mapping and document requests. Compare the string-only and flattened alternatives, then verify that dotted tag keys no longer cause document parsing or mapping conflicts, updating the add cloud metadata documentation if that is the chosen resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, elasticsearch, go
- Domain
- cloud, observability-sre, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100