Kernel log collection stuck after node reboot
- Dominant language
- Go
- Stars
- 28
- Forks
- 17
- Avg merge
- 11h 40m
- Merged PRs (30d)
- 42
Description
Our infrastructure management procedure sometimes involves a node reboot. We have observed that our kernel collection configuration:
```toml
kernel-target = [
{ database = 'Logs', table = 'Kernel', priority = 'info' }
]
```
sometimes (but not always) stops progressing after a node reboot. A mitigation that works is the following steps:
1. Remove the file matching `/mnt/collector/kernel_*`
2. Restart the collector daemon
We are also independently looking at the solution mentioned in https://github.com/Azure/adx-mon/pull/653#pullrequestreview-2734190864.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the kernel-target collection stall across a node reboot with the configuration shown, then inspect the collector's persisted state under /mnt/collector/kernel_*. Review the solution discussed in pull request 653 and compare it with the restart and file-removal workaround. Done means kernel collection resumes after reboot without manually deleting the file or restarting the daemon.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100