kubernetes / kubernetes/node-problem-detector

Gauge for FilesystemIsReadOnly not downgraded to 0 after fixing the problem

Open
#474 31 comments 9 reactions 0 assignees View on GitHub
lifecycle/stale
Dominant language
Go
Stars
3.5k
Forks
702
Avg merge
1d 6h
Merged PRs (30d)
11

Description

The problem occurred when filesystem went to read only mode. That was fixed, but still in the metrics I was able to see the counter and gauge set up to 1.
I conducted a test and multiple times injected the FileSystemIsReadOnly to the /dev/kmsg (https://github.com/kubernetes/node-problem-detector/blob/master/config/kernel-monitor.json):

1 log_monitor.go:160] New status generated: &{Source:kernel-monitor Events:[{Severity:info Timestamp:2020-10-08 06:44:16.09315274 +0000 UTC m=+1331754.148888064 Reason:FilesystemIsReadOnly Message:Node condition ReadonlyFilesystem is now: True, reason: FilesystemIsReadOnly}] Conditions:[{Type:KernelDeadlock Status:False Transition:2020-09-22 20:48:21.98500453 +0000 UTC m=+0.040739839 Reason:KernelHasNoDeadlock Message:kernel has no deadlock} {Type:ReadonlyFilesystem Status:True Transition:2020-10-08 06:44:16.09315274 +0000 UTC m=+1331754.148888064 Reason:FilesystemIsReadOnly Message:Remounting filesystem read-only}]}

Still the metrics were shown as 1 and it did not downgraded to 0. Even the the issue with ro filesystem was fixed, still the metric was 1:

problem_counter{reason="FilesystemIsReadOnly"} 1
problem_gauge{reason="FilesystemIsReadOnly",type="ReadonlyFilesystem"} 1

As a workaround the pod was deleted and after that metrics were reset to 0.
What is the reason of that behaviour? The type "permanent"? Is deleting a pod the only solution?

kernel-monitor.json

{
"type": "permanent",
"condition": "ReadonlyFilesystem",
"reason": "FilesystemIsReadOnly",
"pattern": "Remounting filesystem read-only"
}

Contributor guide

Open the contributing guide

Research direction

Start with config/kernel-monitor.json and log_monitor.go around line 160, then trace how the permanent FilesystemIsReadOnly condition updates problem_counter and problem_gauge. Reproduce the issue by injecting the stated message into /dev/kmsg and compare metrics before and after the filesystem is fixed. Done means the expected reset behavior for the permanent condition is documented or corrected without deleting the pod.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.