sysbox-mgr failed to start
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
We had a node where sysbox-mgr failed to start resulting in no container able to launch with sysbox.
It looks like sysbox-mgr failed to start on time, I attached the logs but they don't tell me much more.
Is there anything else where I can look for more info?
Also would it make sense to have this systemd service restart when failing? I could make a PR for that.
logs:
root@ip-10-0-1-129:/# journalctl -u sysbox-mgr
-- Logs begin at Wed 2023-06-21 09:32:47 UTC, end at Wed 2023-06-21 11:04:30 UTC. --
Jun 21 09:33:54 ip-172-31-47-216 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Starting ..."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Sysbox data root: /var/lib/sysbox"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs module found in kernel: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs works properly: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs-on-overlayfs works properly: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="ID-mapped mounts supported by kernel: yes"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Operating in system container mode."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Inner container image preloading enabled."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Ready ..."
Jun 21 09:33:54 ip-172-31-47-216 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: Stopping sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg="Caught OS signal: terminated"
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg="Stopping (gracefully) ..."
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg=Stopped.
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg=Exiting.
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: sysbox-mgr.service: Succeeded.
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: Stopped sysbox-mgr (part of the Sysbox container runtime).
-- Reboot --
Jun 21 10:45:27 ip-10-0-1-129 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 10:45:27 ip-10-0-1-129 sysbox-mgr[552]: time="2023-06-21 10:45:27" level=info msg="Starting ..."
Jun 21 10:45:27 ip-10-0-1-129 sysbox-mgr[552]: time="2023-06-21 10:45:27" level=info msg="Sysbox data root: /var/lib/sysbox"
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: sysbox-mgr.service: start operation timed out. Terminating.
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: sysbox-mgr.service: Failed with result 'timeout'.
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: Failed to start sysbox-mgr (part of the Sysbox container runtime).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided journalctl -u sysbox-mgr output and inspect the sysbox-mgr systemd service configuration, focusing on the startup timeout. Determine why startup can exceed the timeout and whether restart-on-failure behavior is appropriate; done means the failure is explained and the service behavior is defined or documented.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100