Multiple identical entries in Filebeat registry file

# Description
We deployed the official chart package via Helm, with version 8.5.1. We discovered that the registry file /usr/share/filebeat/data/registry/filebeat/log.json in the Filebeat pod contains tens of thousands of records, and it continues to grow. The vast majority of these records are duplicates, sharing the same combination of device and inode, identical offset, and all other fields are exactly the same.

We suspect that this issue is also the reason why Filebeat is consuming excessively high CPU (200%).

# filebeat.yml
```yaml
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*base-platform-prd-ireland*.log
- /var/log/containers/*life-pre-ireland*.log
exclude_files:
- bp-web-newbie.*
multiline:
pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
negate: true
match: after
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: container
paths:
- /var/log/containers/bp-web-newbie*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: container
paths:
- /var/log/containers/*.log
exclude_files:
- .*base-platform-prd-ireland.*
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"

output.kafka:
hosts: ["..FILTERED.."]
username: kafka
password: [[FILTERED]]
sasl.mechanism: SCRAM-SHA-512
topic: 'filebeat-life'
partition.round_robin:
reachable_only: false
compression: gzip
max_message_bytes: 1000000
max_retries: 3
client_id: 'filebeat-life'
required_acks: 1
topic_metadata:
refresh_frequency: 1m
number_of_partitions: 3
replication_factor: 3
ssl.enabled: true
ssl.verification_mode: none
```

# some lines from /usr/share/filebeat/data/registry/filebeat/log.json
```
{"k":"filebeat::logs::native::206569662-66305","v":{"source":"/var/log/containers/bp-auth-user-644b6586bb-gfd4l_base-platform-prd-ireland_bp-auth-user-2b7950e3813cfadd9b500ceb9abcc9815700e5eca775a6d105ca5ec33d58795f.log","type":"container","FileStateOS":{"device":66305,"inode":206569662},"identifier_name":"native","id":"native::206569662-66305","prev_id":"","offset":2448063,"timestamp":[743145824,1743650311],"ttl":-1}}
{"k":"filebeat::logs::native::206569662-66305","v":{"FileStateOS":{"inode":206569662,"device":66305},"identifier_name":"native","prev_id":"","source":"/var/log/containers/bp-auth-user-644b6586bb-gfd4l_base-platform-prd-ireland_bp-auth-user-2b7950e3813cfadd9b500ceb9abcc9815700e5eca775a6d105ca5ec33d58795f.log","timestamp":[743145824,1743650311],"ttl":-1,"id":"native::206569662-66305","offset":2448063,"type":"container"}}
{"k":"filebeat::logs::native::206569662-66305","v":{"id":"native::206569662-66305","prev_id":"","source":"/var/log/containers/bp-auth-user-644b6586bb-gfd4l_base-platform-prd-ireland_bp-auth-user-2b7950e3813cfadd9b500ceb9abcc9815700e5eca775a6d105ca5ec33d58795f.log","ttl":-1,"type":"container","offset":2448063,"timestamp":[743145824,1743650311],"FileStateOS":{"inode":206569662,"device":66305},"identifier_name":"native"}}
{"k":"filebeat::logs::native::206569662-66305","v":{"prev_id":"","source":"/var/log/containers/bp-auth-user-644b6586bb-gfd4l_base-platform-prd-ireland_bp-auth-user-2b7950e3813cfadd9b500ceb9abcc9815700e5eca775a6d105ca5ec33d58795f.log","timestamp":[743145824,1743650311],"ttl":-1,"type":"container","identifier_name":"native","id":"native::206569662-66305","offset":2448063,"FileStateOS":{"inode":206569662,"device":66305}}}
{"k":"filebeat::logs::native::206569662-66305","v":{"ttl":-1,"prev_id":"","offset":2448063,"timestamp":[743145824,1743650311],"type":"container","FileStateOS":{"device":66305,"inode":206569662},"identifier_name":"native","id":"native::206569662-66305","source":"/var/log/containers/bp-auth-user-644b6586bb-gfd4l_base-platform-prd-ireland_bp-auth-user-2b7950e3813cfadd9b500ceb9abcc9815700e5eca775a6d105ca5ec33d58795f.log"}}
```
# /usr/share/filebeat/logs/filebeat-20250403-1231.ndjson
```
{"log.level":"info","@timestamp":"2025-04-03T03:25:10.247Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-04-03T03:25:10.247Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: c08f69e9-d631-4264-8fbe-6708d515d168","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-04-03T03:25:10.248Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-04-03T03:25:10.248Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":104},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
```

For confirmed bugs, please report:
- Version: 8.5.1
- Operating System: Ubuntu 20.04.5 LTS (container os)
- Discuss Forum URL:
- Steps to Reproduce:
(1) helm install
```bash
kubectl create ns logging
helm -n logging upgrade --install filebeat elastic/filebeat --version 8.5.1 -f values-filebeat.yaml
```
values-filebeat.yaml:
```yaml
---
daemonset:
enabled: true
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*base-platform-prd-ireland*.log
- /var/log/containers/*life-pre-ireland*.log
exclude_files:
- bp-web-newbie.*
multiline:
pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
negate: true
match: after
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: container
paths:
- /var/log/containers/bp-web-newbie*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- type: container
paths:
- /var/log/containers/*.log
exclude_files:
- .*base-platform-prd-ireland.*
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"

output.kafka:
# 初始连接的 Kafka 代理列表
hosts: ["..."]

# 认证
username: kafka
password: ...
sasl.mechanism: SCRAM-SHA-512

# 要写入的 Kafka 主题
topic: 'filebeat-life'

# 可选：设置分区策略
partition.round_robin:
reachable_only: false

# 可选：启用压缩
compression: gzip

# 单条日志大小小于1M
max_message_bytes: 1000000

# 可选：设置最大重试次数
max_retries: 3

# 可选：设置客户端 ID
client_id: 'filebeat-life'

# 可选：设置所需的 ACKs
required_acks: 1

topic_metadata:
refresh_frequency: 1m
# 设置分片数和副本数
number_of_partitions: 3
replication_factor: 3

# 可选：启用 SSL/TLS
ssl.enabled: true
ssl.verification_mode: none

resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "1000m"
memory: "200Mi"
tolerations:
- key: "dedicated"
operator: "Equal"
value: "elasticsearch"
effect: "NoSchedule"
secretMounts:
- name: elasticsearch-master-certs
secretName: cert-tls-es-logging
path: /usr/share/filebeat/certs/
# Root directory where Filebeat will write data to in order to persist registry
# data across pod restarts (file position and other metadata).
hostPathRoot: /var/lib
image: "...."
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
imagePullSecrets:
- name: "regcred-tr"
```
and the filesystem format of k8s worker root path is xfs.

Start with the supplied Helm configuration in values-filebeat.yaml and reproduce the installation using Filebeat 8.5.1. Inspect /usr/share/filebeat/data/registry/filebeat/log.json alongside the Filebeat NDJSON logs, focusing on why identical registry records accumulate. Done means the supplied setup no longer produces duplicate registry entries and the reported CPU behavior is verified.

Written by the indexing model from the issue text.