[KSPM EKS] Cloudbeat can't read kubelet process
- Dominant language
- Go
- Stars
- 58
- Forks
- 55
- Avg merge
- 9h 9m
- Merged PRs (30d)
- 424
Description
### Background
An [alert](https://elastic.slack.com/archives/C056S4ZJW3W/p1709243707032229) was triggered for matching 400 findings instead of 411 for KSPM on EKS.
### Research
1. As you can see in the [findings index](https://live-environment-do-not-delete-stack.kb.us-west2.gcp.elastic-cloud.com:9243/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:'2024-02-28T22:00:00.000Z',to:'2024-03-01T22:00:00.000Z'))&_a=(columns:!(rule.benchmark.rule_number,resource.name,resource.id,host.name),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,key:resource.name,negate:!f,params:(query:kubelet),type:phrase),query:(match_phrase:(resource.name:kubelet))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,key:host.name,negate:!f,params:(query:ip-10-0-3-115.eu-west-1.compute.internal),type:phrase),query:(match_phrase:(host.name:ip-10-0-3-115.eu-west-1.compute.internal)))),index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,interval:auto,query:(language:kuery,query:'%22rule.benchmark.id%22:%20%22cis_eks%22%20'),sort:!(!('@timestamp',asc)))), the 11 missing findings used to come from `ip-10-0-3-115.eu-west-1.compute.internal` node as cloudbeat analyses its `kubelet` process but for some reason this time it failed.
2. When checking cloudbeat [logs]( https://live-environment-do-not-delete-stack.kb.us-west2.gcp.elastic-cloud.com:9243/app/logs/stream?logPosition=(position:(tiebreaker:239858,time:'2024-02-29T20:32:37.274Z'))&logFilter=(filters:!(),query:(language:kuery,query:'elastic_agent.id:aec0e673-00e0-4a53-92da-d001d246ca98%20and%20(data_stream.dataset:elastic_agent.cloudbeat)'),refreshInterval:(pause:!t,value:5000),timeRange:(from:'2024-02-29T20:00:00.000Z',to:'2024-02-29T21:00:00.000Z'))&logView=(logViewId:default,type:log-view-reference)&flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n) ) from the relevant time we can see that cloudbeat failed to read the process from the filesystem: `Error running fetcher for key process: open /hostfs/proc/2626/stat: no such file or directory`.
### Disclaimer
On the next cycle, things got back to normal, cloudbeat properly read and analyzed the kubelet process and generated the missing 11 findings.
### Next steps
We don't have enough data to understand what caused this error I suggest seeing if this issue happens again and prioritizing it accordingly.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.