elastic / elastic/cloudbeat

[KSPM EKS] Cloudbeat can't read kubelet process

Open
#2,008 1 comment 0 reactions 0 assignees View on GitHub
bug cloudbeat Team:Cloud Security
Dominant language
Go
Stars
58
Forks
55
Avg merge
9h 9m
Merged PRs (30d)
424

Description

### Background
An [alert](https://elastic.slack.com/archives/C056S4ZJW3W/p1709243707032229) was triggered for matching 400 findings instead of 411 for KSPM on EKS.

### Research
1. As you can see in the [findings index](https://live-environment-do-not-delete-stack.kb.us-west2.gcp.elastic-cloud.com:9243/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:'2024-02-28T22:00:00.000Z',to:'2024-03-01T22:00:00.000Z'))&_a=(columns:!(rule.benchmark.rule_number,resource.name,resource.id,host.name),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,key:resource.name,negate:!f,params:(query:kubelet),type:phrase),query:(match_phrase:(resource.name:kubelet))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,key:host.name,negate:!f,params:(query:ip-10-0-3-115.eu-west-1.compute.internal),type:phrase),query:(match_phrase:(host.name:ip-10-0-3-115.eu-west-1.compute.internal)))),index:cloud_security_posture-9129a080-7f48-11ec-8249-431333f83c5f,interval:auto,query:(language:kuery,query:'%22rule.benchmark.id%22:%20%22cis_eks%22%20'),sort:!(!('@timestamp',asc)))), the 11 missing findings used to come from `ip-10-0-3-115.eu-west-1.compute.internal` node as cloudbeat analyses its `kubelet` process but for some reason this time it failed.
2. When checking cloudbeat [logs]( https://live-environment-do-not-delete-stack.kb.us-west2.gcp.elastic-cloud.com:9243/app/logs/stream?logPosition=(position:(tiebreaker:239858,time:'2024-02-29T20:32:37.274Z'))&logFilter=(filters:!(),query:(language:kuery,query:'elastic_agent.id:aec0e673-00e0-4a53-92da-d001d246ca98%20and%20(data_stream.dataset:elastic_agent.cloudbeat)'),refreshInterval:(pause:!t,value:5000),timeRange:(from:'2024-02-29T20:00:00.000Z',to:'2024-02-29T21:00:00.000Z'))&logView=(logViewId:default,type:log-view-reference)&flyoutOptions=(flyoutId:!n,flyoutVisibility:hidden,surroundingLogsId:!n) ) from the relevant time we can see that cloudbeat failed to read the process from the filesystem: `Error running fetcher for key process: open /hostfs/proc/2626/stat: no such file or directory`.

### Disclaimer
On the next cycle, things got back to normal, cloudbeat properly read and analyzed the kubelet process and generated the missing 11 findings.

### Next steps
We don't have enough data to understand what caused this error I suggest seeing if this issue happens again and prioritizing it accordingly.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.