elastic / elastic/beats

Improve scalability of Kubernetes module in metricbeat

Open
#32,662 0 comments 0 reactions 1 assignee View on GitHub

@gsantoro is already working on this.

Since Aug 11, 2022.

Team:Cloudnative-Monitoring
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

Context
During my investigation for the the PR 32539 I have noticed that there might be room for performance improvements in the Kubernetes module of metricbeat.

Each metricbeat instance is storing metrics about all the nodes in the Kubernetes cluster but only metrics about pods and containers on the same node where that instance of metricbeat is running. This replicates how the previous expiring cache worked but it is now more evident and can have detrimental effect in clusters with lots of nodes. This is because, with lots of nodes we might end up wasting lots of memory on unused metrics from other nodes. This behaviour is due to how the watcher notifies events from Kubernetes and it wasn't modified by the afore mentioned PR.

Possible solution is for each metricbeat to filter out events generated by other nodes than the one where it is running. This should simplify the MetricRepo API since we wouldn't need to handle the deletion of nodes but only events from Pods and Containers.

During the same investigation, I noticed that when a Pod is deleted, it first calls the update function (to add its metrics again) to be deleted few seconds after. I am not sure if this is intended since the status of the pod is Terminating already. Also I noticed that the call to deletePod is executing twice. This might be because there is more than 1 watcher or because the code is shared between multiple metricsets.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.