Docker swarm join-token empty
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.7k
- Forks
- 676
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
I ran into a really weird issue last night with our docker swarm instance. One of our servers filled its disk space with logs which caused one of our servers to fully fill up (this caused the issue). Once that happened, our entire docker swarm system stopped responding, all workers and managers to a point they would not communicate with each other and I kept getting the error not enough managers online. I checked to make sure the managers were all online and could communicate and they could. At that point I figured it would probably be good to just reset the quorum, so I ran the command docker swarm init --force-new-cluster --advertise-addr node01:2377 (with the proper credentials). It said it was successful and to promote other workers to managers, but it would not let me. When I ran the command docker swarm join-token worker, nothing was returned. It is like the key used to connect all the servers to the swarm just disappeared. In the end I had to clear our the swarm completely and re-build every single service. Has anyone experienced this before? I still have a few servers from the old swarm if more info is needed to help debug this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Docker Swarm recovery commands described here: docker swarm init --force-new-cluster and docker swarm join-token worker. Investigate the quorum-reset path and the conditions that could leave the join token empty, using the remaining old-swarm servers if reproducible. Done means the cause is identified and a verified recovery or diagnostic result is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100