Azure / Azure/adx-mon

Kernel log collection stuck after node reboot

Open
#1,048 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
28
Forks
17
Avg merge
11h 40m
Merged PRs (30d)
42

Description

Our infrastructure management procedure sometimes involves a node reboot. We have observed that our kernel collection configuration:

```toml
kernel-target = [
{ database = 'Logs', table = 'Kernel', priority = 'info' }
]
```

sometimes (but not always) stops progressing after a node reboot. A mitigation that works is the following steps:

1. Remove the file matching `/mnt/collector/kernel_*`
2. Restart the collector daemon

We are also independently looking at the solution mentioned in https://github.com/Azure/adx-mon/pull/653#pullrequestreview-2734190864.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the kernel-target collection stall across a node reboot with the configuration shown, then inspect the collector's persisted state under /mnt/collector/kernel_*. Review the solution discussed in pull request 653 and compare it with the restart and file-removal workaround. Done means kernel collection resumes after reboot without manually deleting the file or restarting the daemon.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.