nestybox / nestybox/sysbox

sysbox-mgr failed to start

Open
#713 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

We had a node where sysbox-mgr failed to start resulting in no container able to launch with sysbox.

It looks like sysbox-mgr failed to start on time, I attached the logs but they don't tell me much more.
Is there anything else where I can look for more info?

Also would it make sense to have this systemd service restart when failing? I could make a PR for that.

logs:

root@ip-10-0-1-129:/# journalctl -u sysbox-mgr
-- Logs begin at Wed 2023-06-21 09:32:47 UTC, end at Wed 2023-06-21 11:04:30 UTC. --
Jun 21 09:33:54 ip-172-31-47-216 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Starting ..."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Sysbox data root: /var/lib/sysbox"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs module found in kernel: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs works properly: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Shiftfs-on-overlayfs works properly: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="ID-mapped mounts supported by kernel: yes"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Operating in system container mode."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Inner container image preloading enabled."
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Jun 21 09:33:54 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:33:54" level=info msg="Ready ..."
Jun 21 09:33:54 ip-172-31-47-216 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: Stopping sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg="Caught OS signal: terminated"
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg="Stopping (gracefully) ..."
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg=Stopped.
Jun 21 09:47:18 ip-172-31-47-216 sysbox-mgr[6506]: time="2023-06-21 09:47:18" level=info msg=Exiting.
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: sysbox-mgr.service: Succeeded.
Jun 21 09:47:18 ip-172-31-47-216 systemd[1]: Stopped sysbox-mgr (part of the Sysbox container runtime).
-- Reboot --
Jun 21 10:45:27 ip-10-0-1-129 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Jun 21 10:45:27 ip-10-0-1-129 sysbox-mgr[552]: time="2023-06-21 10:45:27" level=info msg="Starting ..."
Jun 21 10:45:27 ip-10-0-1-129 sysbox-mgr[552]: time="2023-06-21 10:45:27" level=info msg="Sysbox data root: /var/lib/sysbox"
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: sysbox-mgr.service: start operation timed out. Terminating.
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: sysbox-mgr.service: Failed with result 'timeout'.
Jun 21 10:46:12 ip-10-0-1-129 systemd[1]: Failed to start sysbox-mgr (part of the Sysbox container runtime).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided journalctl -u sysbox-mgr output and inspect the sysbox-mgr systemd service configuration, focusing on the startup timeout. Determine why startup can exceed the timeout and whether restart-on-failure behavior is appropriate; done means the failure is explained and the service behavior is defined or documented.

Written by the indexing model from the issue text.

Assessment

Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.