umil production needs reboot frequently
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
I'm not sure what happens, but it seems that there's something that fails often, and the only way to fix it is to reboot the server.
Some recurrent logs:
```
Jul 14 15:41:20 prod-vim kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 15:41:20 prod-vim kernel: INFO: task fwupd:4158190 blocked for more than 120 seconds.
Jul 14 15:41:20 prod-vim kernel: Not tainted 5.4.0-216-generic #236-Ubuntu
Jul 14 15:41:20 prod-vim kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 03:59:35 prod-vim systemd[1]: snapd.service: Watchdog timeout (limit 5min)!
Jul 16 04:03:19 prod-vim systemd[1]: Failed to start Snap Daemon.
Jul 16 04:04:50 prod-vim systemd[1]: Failed to start Snap Daemon.
Jul 16 04:06:21 prod-vim systemd[1]: Failed to start Snap Daemon.
...
Jul 28 06:32:06 prod-vim systemd[1]: Failed to start Snap Daemon.
Jul 28 06:33:37 prod-vim systemd[1]: Failed to start Snap Daemon.
Jul 28 06:35:09 prod-vim systemd[1]: Failed to start Snap Daemon.
Jul 29 10:10:09 prod-vim systemd[1]: snapd.service: Watchdog timeout (limit 5min)!
```
#### 1. Hung Task Detected
```
INFO: task fwupd:4158190 blocked for more than 120 seconds.
```
This means the `fwupd` (Firmware Update Daemon) process was stuck (blocked on I/O or waiting on a resource) for over 2 minutes. It suggests possible disk I/O issues, filesystem problems, or kernel bugs. This can cause cascading issues where other processes start hanging or timing out.
#### 2. snapd.service: Watchdog timeout
```
snapd.service: Watchdog timeout (limit 5min)!
Failed to start Snap Daemon.
```
The `snapd` service (which manages Snap packages) hung and exceeded its 5-minute watchdog timeout. It repeatedly failed to start, possibly because of the same I/O or blocking issue.
I checked other logs and confirmed that there are no disk I/O errors or No OOM kills.
Possible fix:
1. Disable `fwupd` by `sudo systemctl disable --now fwupd.service fwupd-refresh.timer`
2. Disable `snapd`
However, we are using `snapd` as follows
```
Name Version Rev Tracking Publisher Notes
core20 20250526 2599 latest/stable canonical✓ base
lxd 4.0.10-d342340 32662 4.0/stable/… canonical✓ -
snapd 2.70 24792 latest/stable canonical✓ snapd
```
In the short term, we might try `sudo systemctl restart snapd` for a quick fix. Or we can make `snapd` restart automatically.
Still not sure if this is the main cause but hopefully this is related. @notkaramel
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.