error on running swarm [manager stopped: can't initialize raft node: WAL error cannot be repaired: unexpected EOF]
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
The problem happened after the machine run out of space.
Now I cannot leave the swarm either create new containers.
docker service ls
Error response from daemon: This node is not a swarm manager. Worker nodes can't be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.
manager node
docker info
Containers: 35
Running: 14
Paused: 0
Stopped: 21
Images: 77
Server Version: 17.09.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: error
NodeID:
Error: manager stopped: can't initialize raft node: WAL error cannot be repaired: unexpected EOF
Is Manager: false
Node Address: 10.10.10.62
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 46.77GiB
Name: BJ-H03-12-cm.getui
ID: 34OK:O5JK:V3PU:SMDX:6SJS:ZT76:CIZ4:AHX7:OKAT:U2SK:LFGR:7T2S
Docker Root Dir: /app/docker/dataroot
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://kohnnhik.mirror.aliyuncs.com/
Live Restore Enabled: false
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
WARNING: bridge-nf-call-ip6tables is disabled
I've tried to leave the swarm but it hasn't worked:
$ docker swarm leave
Error response from daemon: context deadline exceeded
$ docker swarm init
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
$ docker swarm leave --force
Error response from daemon: context deadline exceeded
Similar and unsolved: https://github.com/docker/classicswarm/issues/2819
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported docker swarm and docker info output, focusing on manager initialization and the Raft WAL unexpected EOF error. Reproduce the failure after the machine ran out of space and compare docker swarm leave, docker swarm leave --force, and docker swarm init. Done means the manager initializes and the swarm can be left or re-created without these errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100