add_fields processor: does not overwrite the target field if it already exists
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
I am using `add_fields` processor to add `orchestrator.cluster.name`
```
- add_fields:
target: orchestrator.cluster
fields:
name: "test"
```
Note: it is a way we mention in our public doc - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_dashboard_32 for cases when cluster.name can't be detected automatically.
When testing (on kind cluster, where orchestrator.cluster.name can be defined) I've noticed that the field value is added to the existing field, but not get overwrite as claimed in [doc](https://www.elastic.co/guide/en/beats/metricbeat/current/add-fields.html) and suppose to be in [tests](https://github.com/elastic/beats/blob/v8.15.1/libbeat/processors/actions/add_fields_test.go#L45-L51):

8.16:
```
{
"_index": ".ds-metricbeat-8.16.0-2024.09.11-000001",
"_id": "40if4ZEBJGFURqBQZ8MC",
"_version": 1,
"_score": 0,
"_source": {
"@timestamp": "2024-09-11T15:06:45.113Z",
"metricset": {
"name": "node",
"period": 10000
},
"kubernetes": {
"node": {
"memory": {
"usage": {
"bytes": 4808962048
},
"workingset": {
"bytes": 2266112000
},
"rss": {
"bytes": 1402314752
},
"pagefaults": 57223252,
"majorpagefaults": 1610,
"available": {
"bytes": 14503260160
}
},
"network": {
"tx": {
"bytes": 596430470,
"errors": 0
},
"rx": {
"bytes": 1133629283,
"errors": 0
}
},
"start_time": "2024-08-26T07:48:42Z",
"name": "kind-control-plane",
"fs": {
"capacity": {
"bytes": 369759891456
},
"used": {
"bytes": 219972317184
},
"inodes": {
"used": 3508210,
"free": 19445774,
"count": 22953984
},
"available": {
"bytes": 130970845184
}
},
"runtime": {
"imagefs": {
"available": {
"bytes": 130970845184
},
"capacity": {
"bytes": 369759891456
},
"used": {
"bytes": 3731861504
}
}
},
"cpu": {
"usage": {
"nanocores": 229101086,
"core": {
"ns": 49213133124000
}
}
}
},
"labels": {
"beta_kubernetes_io/os": "linux",
"kubernetes_io/arch": "amd64",
"kubernetes_io/hostname": "kind-control-plane",
"kubernetes_io/os": "linux",
"node-role_kubernetes_io/control-plane": "",
"beta_kubernetes_io/arch": "amd64"
}
},
"agent": {
"ephemeral_id": "cbe5bf1e-f613-4c77-9517-530e189263f5",
"id": "c6bad5b7-9f96-4701-812e-bd08be0e0d39",
"name": "kind-control-plane",
"type": "metricbeat",
"version": "8.16.0"
},
"orchestrator.cluster": {
"name": "test"
},
"service": {
"address": "https://kind-control-plane:10250/stats/summary",
"type": "kubernetes"
},
"event": {
"dataset": "kubernetes.node",
"module": "kubernetes",
"duration": 554803
},
"host": {
"name": "kind-control-plane"
},
"orchestrator": {
"cluster": {
"url": "kind-control-plane:6443",
"name": "kind"
}
},
"ecs": {
"version": "8.0.0"
}
},
"fields": {
"orchestrator.cluster.name": [
"kind",
"test"
],
"kubernetes.labels.kubernetes_io/arch": [
"amd64"
],
"kubernetes.node.runtime.imagefs.available.bytes": [
130970845184
],
"service.type": [
"kubernetes"
],
"kubernetes.labels.beta_kubernetes_io/arch": [
"amd64"
],
"kubernetes.labels.kubernetes_io/hostname": [
"kind-control-plane"
],
"agent.name": [
"kind-control-plane"
],
"host.name": [
"kind-control-plane"
],
"beats_state.state.host.name": [
"kind-control-plane"
],
"kubernetes.node.runtime.imagefs.capacity.bytes": [
369759891456
],
"agent.hostname": [
"kind-control-plane"
],
"agent.id": [
"c6bad5b7-9f96-4701-812e-bd08be0e0d39"
],
"ecs.version": [
"8.0.0"
],
"service.address": [
"https://kind-control-plane:10250/stats/summary"
],
"agent.version": [
"8.16.0"
],
"kubernetes.node.fs.inodes.used": [
3508210
],
"kubernetes.node.network.rx.bytes": [
1133629283
],
"kubernetes.node.network.tx.bytes": [
596430470
],
"kubernetes.node.fs.inodes.count": [
22953984
],
"kubernetes.node.memory.majorpagefaults": [
1610
],
"kubernetes.node.fs.inodes.free": [
19445774
],
"kubernetes.node.name": [
"kind-control-plane"
],
"kubernetes.node.runtime.imagefs.used.bytes": [
3731861504
],
"kubernetes.labels.beta_kubernetes_io/os": [
"linux"
],
"kubernetes.labels.node-role_kubernetes_io/control-plane": [
""
],
"kubernetes.labels.kubernetes_io/os": [
"linux"
],
"kubernetes.node.memory.pagefaults": [
57223252
],
"kubernetes.node.start_time": [
"2024-08-26T07:48:42.000Z"
],
"kubernetes.node.fs.used.bytes": [
219972317184
],
"kubernetes.node.memory.rss.bytes": [
1402314752
],
"agent.type": [
"metricbeat"
],
"kubernetes.node.cpu.usage.core.ns": [
49213133124000
],
"orchestrator.cluster.url": [
"kind-control-plane:6443"
],
"logstash_stats.timestamp": [
"2024-09-11T15:06:45.113Z"
],
"event.module": [
"kubernetes"
],
"kubernetes.node.cpu.usage.nanocores": [
229101086
],
"beats_state.timestamp": [
"2024-09-11T15:06:45.113Z"
],
"timestamp": [
"2024-09-11T15:06:45.113Z"
],
"kubernetes.node.network.tx.errors": [
0
],
"kubernetes.node.memory.usage.bytes": [
4808962048
],
"kibana_stats.timestamp": [
"2024-09-11T15:06:45.113Z"
],
"kubernetes.node.fs.available.bytes": [
130970845184
],
"metricset.period": [
10000
],
"kubernetes.node.network.rx.errors": [
0
],
"kubernetes.node.memory.available.bytes": [
14503260160
],
"event.duration": [
554803
],
"metricset.name": [
"node"
],
"kubernetes.node.fs.capacity.bytes": [
369759891456
],
"kubernetes.node.memory.workingset.bytes": [
2266112000
],
"@timestamp": [
"2024-09-11T15:06:45.113Z"
],
"agent.ephemeral_id": [
"cbe5bf1e-f613-4c77-9517-530e189263f5"
],
"event.dataset": [
"kubernetes.node"
]
}
}
```
specifically notice here:
```
"_source": {
...
"orchestrator.cluster": {
"name": "test"
},
...
"orchestrator": {
"cluster": {
"url": "kind-control-plane:6443",
"name": "kind"
}
},
...
},
"fields": {
"orchestrator.cluster.name": [
"kind",
"test"
],
```
That behavior might have changed after this PR: https://github.com/elastic/beats/pull/14407
```
{
"_index": "metricbeat-7.5.2-2024.09.11-000001",
"_id": "m1B15ZEBJGFURqBQ7sbZ",
"_version": 1,
"_score": 0,
"_source": {
"@timestamp": "2024-09-12T08:59:55.997Z",
"service": {
"address": "kind-control-plane:10250",
"type": "kubernetes"
},
"kubernetes": {
"pod": {
"name": "elastic-agent-llh2m",
"uid": "47e3d2a2-c3c9-4fad-9d36-ad74ed7f0548"
},
"labels": {
"controller-revision-hash": "6fd88bf58b",
"pod-template-generation": "10",
"app": "elastic-agent"
},
"container": {
"memory": {
"usage": {
"limit": {
"pct": 0.03879597028395844
},
"bytes": 650584064,
"node": {
"pct": 0.03879597028395844
}
},
"workingset": {
"bytes": 650584064
},
"rss": {
"bytes": 617177088
},
"pagefaults": 18396612,
"majorpagefaults": 33,
"available": {
"bytes": 83419136
}
},
"rootfs": {
"available": {
"bytes": 128244932608
},
"capacity": {
"bytes": 369759891456
},
"used": {
"bytes": 62169088
},
"inodes": {
"used": 69
}
},
"logs": {
"available": {
"bytes": 128244932608
},
"capacity": {
"bytes": 369759891456
},
"used": {
"bytes": 13045760
},
"inodes": {
"used": 5,
"free": 19422833,
"count": 22953984
}
},
"start_time": "2024-09-10T16:03:45Z",
"name": "elastic-agent",
"cpu": {
"usage": {
"nanocores": 19947667,
"core": {
"ns": 2722178877000
},
"node": {
"pct": 0.0012467291875
},
"limit": {
"pct": 0.0012467291875
}
}
}
},
"namespace": "kube-system",
"node": {
"name": "kind-control-plane"
}
},
"event": {
"dataset": "kubernetes.container",
"module": "kubernetes",
"duration": 32475075
},
"agent": {
"hostname": "kind-control-plane",
"id": "c6bad5b7-9f96-4701-812e-bd08be0e0d39",
"version": "7.5.2",
"type": "metricbeat",
"ephemeral_id": "a6b39abc-f565-4597-a80c-d5434f13be9d"
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "kind-control-plane"
},
"orchestrator.cluster": {
"name": "test"
},
"metricset": {
"period": 10000,
"name": "container"
}
},
"fields": {
"kubernetes.node.name": [
"kind-control-plane"
],
"orchestrator.cluster.name": [
"test"
],
"kubernetes.container.logs.inodes.free": [
19422833
],
"kubernetes.container.logs.inodes.used": [
5
],
"kubernetes.pod.uid": [
"47e3d2a2-c3c9-4fad-9d36-ad74ed7f0548"
],
"kubernetes.container.logs.used.bytes": [
13045760
],
"kubernetes.container.memory.usage.bytes": [
650584064
],
"kubernetes.container.rootfs.used.bytes": [
62169088
],
"kubernetes.container.start_time": [
"2024-09-10T16:03:45.000Z"
],
"kubernetes.container.cpu.usage.nanocores": [
19947667
],
"service.type": [
"kubernetes"
],
"agent.type": [
"metricbeat"
],
"kubernetes.container.cpu.usage.core.ns": [
2722178877000
],
"event.module": [
"kubernetes"
],
"kubernetes.labels.app": [
"elastic-agent"
],
"kubernetes.container.rootfs.inodes.used": [
69
],
"kubernetes.namespace": [
"kube-system"
],
"kubernetes.container.memory.usage.limit.pct": [
0.039
],
"kubernetes.pod.name": [
"elastic-agent-llh2m"
],
"host.name": [
"kind-control-plane"
],
"kubernetes.container.memory.usage.node.pct": [
0.039
],
"kubernetes.container.memory.majorpagefaults": [
33
],
"kubernetes.container.name": [
"elastic-agent"
],
"kubernetes.container.logs.available.bytes": [
128244932608
],
"kubernetes.container.logs.inodes.count": [
22953984
],
"kubernetes.container.memory.available.bytes": [
83419136
],
"metricset.period": [
10000
],
"kubernetes.container.cpu.usage.node.pct": [
0.001
],
"kubernetes.container.memory.pagefaults": [
18396612
],
"kubernetes.container.rootfs.capacity.bytes": [
369759891456
],
"agent.hostname": [
"kind-control-plane"
],
"kubernetes.labels.pod-template-generation": [
"10"
],
"event.duration": [
32475075
],
"metricset.name": [
"container"
],
"kubernetes.container.cpu.usage.limit.pct": [
0.001
],
"kubernetes.container.logs.capacity.bytes": [
369759891456
],
"@timestamp": [
"2024-09-12T08:59:55.997Z"
],
"agent.id": [
"c6bad5b7-9f96-4701-812e-bd08be0e0d39"
],
"ecs.version": [
"1.1.0"
],
"kubernetes.labels.controller-revision-hash": [
"6fd88bf58b"
],
"service.address": [
"kind-control-plane:10250"
],
"kubernetes.container.rootfs.available.bytes": [
128244932608
],
"agent.ephemeral_id": [
"a6b39abc-f565-4597-a80c-d5434f13be9d"
],
"kubernetes.container.memory.rss.bytes": [
617177088
],
"agent.version": [
"7.5.2"
],
"kubernetes.container.memory.workingset.bytes": [
650584064
],
"event.dataset": [
"kubernetes.container"
]
}
}
```
For confirmed bugs, please report:
- Version: > 7.5.2
- Operating System:
- Discuss Forum URL:
- Steps to Reproduce:
Contributor guide
Research direction
Start with libbeat/processors/actions/add_fields_test.go, especially the referenced lines, and compare its expectations with the add_fields behavior described in the issue. Review the linked add_fields documentation and PR 14407 for the claimed overwrite semantics. Done means an existing target field is overwritten as documented and the relevant test covers the observed case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100