docker pull failed frequently because of netns pollution of docker daemon thread

following https://github.com/docker/libnetwork/issues/1113

we have came across a bunch of failure of docker pull in production, dig deeper and find that https://github.com/docker/libnetwork/issues/1113 is the right answer to this problem,.

ls -al /proc/$(cat /run/docker.pid)/task/*/ns/net I can find threads in docker daemon's thread pool have different netns. most of them are the same with host, some are different.

under such case, a new docker pull may be scheduled to a polluted thread, whose netns is not right, and can not reach hub. This is not easy to reproduce, but it did happen. according to https://github.com/docker/libnetwork/issues/1113, it's related to a golang bug(or not?). Anyway we need to do something to prevent this from happening in production again. Any suggestions on how to solve this in libnetwork? some workaround maybe?

This is about how to reproduce this bug easily:

#!/bin/bash
docker rm -f `docker ps -aq`
ii=0
while true; do
        echo "this is $ii trial"
	ii=$(($ii+1))
	systemctl restart docker

        for i in $(seq 1 5)
        do
         docker run -tid --name test$i busybox sh &
        done
  	cn=`docker ps -q | wc -l`
	while [ "$cn""x" != "7x" ]; do
	  sleep 1;
	  cn=`docker ps -q | wc -l`
        done

        ls -al /proc/$(cat /var/run/docker.pid)/task/*/ns/net > ftest
	cnt=`cat ftest | awk -F : '{print $3}' | sort | uniq | wc -l`
        if [ "$cnt""x" != "1x" ]; then
          echo "fail"
	  cat ftest
	  echo 
          exit 1
        else
          echo "succeed"
        fi

        for i in $(seq 1 5)
        do
	  docker rm -f test$i
        done

done

with this script, I can reproduce this problem(having a thread with different netns in daemon threads pool) within few minutes( maybe tens if unlucky). which means this problem do happen occasionally. And I get a more producible way:

diff --git a/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go b/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go
index f0084b0..a07aff2 100644
--- a/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go
+++ b/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go
@@ -358,7 +358,7 @@ func nsInvoke(path string, prefunc func(nsFD int) error, postfunc func(callerFD
        if err = netns.Set(netns.NsHandle(nsFD)); err != nil {
                return err
        }
-       defer ns.SetNamespace()
+       // defer ns.SetNamespace()
 
        // Invoked after the namespace switch.
        return postfunc(ns.ParseHandlerInt())

performing nsInvoke without setting its namespace back, after creating several containers, you will get daemon with lots of polluted thread. and do docker pull will fail frequently (maybe always).

Start with vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go and the nsInvoke path, then run the provided container/restart loop to observe differing daemon thread network namespaces. Compare behavior with namespace restoration enabled and use the /proc/.../task/*/ns/net checks to confirm the daemon threads remain in one namespace and docker pull no longer fails.