microsoft / microsoft/pai

CPU usage in 192.168.131.186:9100 is above 98%

Open
#5,667 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

这两天收到邮件提醒集群中有个节点CPU占满,并提醒该节点k8sDockerDaemonNotOk
image
image

进入该节点后,使用top命令发现有很多lsof命令在占用cpu
image

使用ps -axjf 发现好像是containerd-shim一直在创建lsof进程。

image

请问这个问题该如何解决
Organization Name:

Short summary about the issue/question:

Brief what process you are following:

How to reproduce it:

OpenPAI Environment:

  • OpenPAI version:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Hardware (e.g. core number, memory size, storage size, GPU type etc.):
  • Others:

Anything else we need to know:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the reported node diagnostics: the high CPU alert, top output, ps -axjf output, and the relationship between containerd-shim and repeated lsof processes. Check the OpenPAI version, operating system, kernel, hardware, and deployment details requested in the issue. Done means identifying the cause and documenting a verified resolution for the affected Kubernetes node.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, linux
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.