moby / moby/swarmkit

error on running swarm [manager stopped: can't initialize raft node: WAL error cannot be repaired: unexpected EOF]

Open
#2,959 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.7k
Forks
676
Avg merge
4d 9h
Merged PRs (30d)
6

Description

The problem happened after the machine run out of space.
Now I cannot leave the swarm either create new containers.

docker service ls
Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.

manager node

docker info
Containers: 35
 Running: 14
 Paused: 0
 Stopped: 21
Images: 77
Server Version: 17.09.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: error
 NodeID:
 Error: manager stopped: can't initialize raft node: WAL error cannot be repaired: unexpected EOF
 Is Manager: false
 Node Address: 10.10.10.62
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 46.77GiB
Name: BJ-H03-12-cm.getui
ID: 34OK:O5JK:V3PU:SMDX:6SJS:ZT76:CIZ4:AHX7:OKAT:U2SK:LFGR:7T2S
Docker Root Dir: /app/docker/dataroot
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://kohnnhik.mirror.aliyuncs.com/
Live Restore Enabled: false

WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
WARNING: bridge-nf-call-ip6tables is disabled

I've tried to leave the swarm but it hasn't worked:

$ docker swarm leave
Error response from daemon: context deadline exceeded
$ docker swarm init
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
$ docker swarm leave --force
Error response from daemon: context deadline exceeded

Similar and unsolved: https://github.com/docker/classicswarm/issues/2819

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported docker swarm and docker info output, focusing on manager initialization and the Raft WAL unexpected EOF error. Reproduce the failure after the machine ran out of space and compare docker swarm leave, docker swarm leave --force, and docker swarm init. Done means the manager initializes and the swarm can be left or re-created without these errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.