Filebeat doesn't collect logs of CronJob pods
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
We experience the same issue mentioned here https://github.com/elastic/beats/issues/22718
We use the latest Filebeat **8.5.1** on AWS EKS with Amazon Linux 2 EC2 worker nodes.
Sometimes Filebeat loses CronJob Pods log mesages.
For CronJobs that runs every minute we see only 54 message with one hour instead of 60.
Around **9-10%** of CronJob messages are lost.
In comparison the legacy Filebeat 6.8.13 doesn't lose any message.
Filebeat version **8.5.1**.
AWS EKS is **v1.21.14**
Filebeat configuration:
```yaml
filebeat.autodiscover:
providers:
- type: kubernetes
cleanup_timeout: 24h
add_resource_metadata:
namespace:
include_labels: [""]
include_annotations: [""]
node:
include_labels: [""]
include_annotations: [""]
deployment: true
cronjob: true
templates:
- config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
```
Cronjob example:
```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: cronjob-test
namespace: default
spec:
schedule: "* * * * *"
successfulJobsHistoryLimit: 30
jobTemplate:
spec:
template:
spec:
containers:
- name: cronjob
image: busybox:1.28
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- date
restartPolicy: OnFailure
```
Example of messages on Kibana.
Here you can see that messages at 15:25 and 15:30 and 15:31 are missing:
```
Dec 14, 2022 @ 15:34:04.596 cronjob-test-27850474-m6pnd Wed Dec 14 14:34:04 UTC 2022
Dec 14, 2022 @ 15:33:04.570 cronjob-test-27850473-bxzdp Wed Dec 14 14:33:04 UTC 2022
Dec 14, 2022 @ 15:32:04.823 cronjob-test-27850472-nrr7p Wed Dec 14 14:32:04 UTC 2022
Dec 14, 2022 @ 15:29:03.972 cronjob-test-27850469-9gbkx Wed Dec 14 14:29:03 UTC 2022
Dec 14, 2022 @ 15:28:03.997 cronjob-test-27850468-5l5c7 Wed Dec 14 14:28:03 UTC 2022
Dec 14, 2022 @ 15:27:03.710 cronjob-test-27850467-24g44 Wed Dec 14 14:27:03 UTC 2022
Dec 14, 2022 @ 15:26:03.730 cronjob-test-27850466-qdxk9 Wed Dec 14 14:26:03 UTC 2022
Dec 14, 2022 @ 15:24:06.376 cronjob-test-27850464-ztclr Wed Dec 14 14:24:06 UTC 2022
Dec 14, 2022 @ 15:23:06.970 cronjob-test-27850463-f2qm6 Wed Dec 14 14:23:06 UTC 2022
```
Contributor guide
Assessment
This issue has not been assessed yet.