moby / moby/libnetwork

Issue with overlay networks and Zookeeper as cluster store

Open
#2,006 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
875
PR merge metrics
No merged PRs in 30d

Description

I set up zookeeper as cluster store, but for some reason, I can't create overlay networks:

$ cat /etc/docker/daemon.json
{
        "cluster-store": "zk://zookeeper.example.com/",
        "cluster-advertise": "eth1:2376",
        "debug": true
}


$ docker info 
...
Cluster Store: zk://zookeeper.example.com/
Cluster Advertise: <ip>:2376
...

$ docker network create -d overlay testnet
Error response from daemon: pool configuration failed because of zk: unknown error

root@db05:~# tailf /var/log/upstart/docker.log
time="2017-10-31T14:38:42.546865312Z" level=debug msg="Calling GET /_ping"
time="2017-10-31T14:38:42.547828124Z" level=debug msg="Calling POST /v1.32/networks/create"
time="2017-10-31T14:38:42.547962672Z" level=debug msg="form data: {\"Attachable\":false,\"CheckDuplicate\":true,\"ConfigFrom\":null,\"ConfigOnly\":false,\"Driver\":\"overlay\",\"EnableIPv6\":false,\"IPAM\":{\"Config\":[],\"Driver\":\"default\",\"Options\":{}},\"Ingress\":false,\"Internal\":false,\"Labels\":{},\"Name\":\"testnet\",\"Options\":{},\"Scope\":\"\"}"
time="2017-10-31T14:38:42.558957407Z" level=debug msg="failed to get networks for scope global: zk: node does not exist"
time="2017-10-31T14:38:42.559239286Z" level=debug msg="Allocating IPv4 pools for network testnet (e7721dc0ed40d6741f6a82fc020cb07f462f1f94afdc5d6ff07baa1a2bd8be47)"
time="2017-10-31T14:38:42.559270326Z" level=debug msg="RequestPool(GlobalDefault, , , map[], false)"
time="2017-10-31T14:38:42.576364230Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: pool configuration failed because of zk: unknown error" error_type=types.internal module=api
time="2017-10-31T14:38:42.576394624Z" level=error msg="Handler for POST /v1.32/networks/create returned error: pool configuration failed because of zk: unknown error"
time="2017-10-31T14:38:42.576418525Z" level=debug msg="FIXME: Got an API for which error does not match any expected type!!!: pool configuration failed because of zk: unknown error" error_type=types.internal module=api

Looks like /docker/network path is missing on Zookeeper:

[zk: localhost:2181(CONNECTED) 9] ls /docker
[nodes]
[zk: localhost:2181(CONNECTED) 10] ls /docker/nodes
[<ip>:2376]

Is there something I can do here? Can I add manually some nodes to zookeeper?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing docker network create -d overlay testnet with the shown daemon.json and inspect the daemon log for the ZooKeeper path errors. Check the /docker and /docker/nodes entries in ZooKeeper and trace the global network lookup; done means identifying the cause of the missing path or documenting the required initialization steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.