CLOSE_WAIT Too many Socket Connections in this state.

Hello All,

We are using version 2.6.x. We use the docker SDK to interfact with docker over TCP (TLS) port 2376. Using the library we do several docker operations like create/start/stop/delete/prune.

We have been noticing that library is leaving the socket connections in CLOSE_WAIT. There is no way we can close these connections unless we restart the service.

Just wanted to check with you all if this is a known issue that got fixed in the later versions. For us moving to later versions is not very straight forward.

We use the docker session via context manager.

@contextmanager
def session_scope(self):
    """Provide a transactional scope around a series of operations."""
    session = None
    try:
        session = self.getClient()
        yield session
    except:
        pass
    finally:
        session.close()


def pruneVolumes(self,**kwargs):
    with self.session_scope() as session:
        return session.prune_volumes(filters=kwargs.get('filters',{}))

After running repeated tests, we are seeing the below.

python 7 164 root 242u IPv4 338932810 0t0 TCP 172.17.0.12:60082->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 243u IPv4 338939148 0t0 TCP 172.17.0.12:47290->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 244u IPv4 338932818 0t0 TCP 172.17.0.12:60096->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 245u IPv4 338939307 0t0 TCP 172.17.0.12:34242->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 246u IPv4 338940450 0t0 TCP 172.17.0.12:34214->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 247u IPv4 338940480 0t0 TCP 172.17.0.12:47518->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 248u IPv4 338946715 0t0 TCP 172.17.0.12:49684->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 249u IPv4 338947187 0t0 TCP 172.17.0.12:36582->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 250u IPv4 338947042 0t0 TCP 172.17.0.12:49922->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 251u IPv4 338947167 0t0 TCP 172.17.0.12:36564->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 252u IPv4 338954212 0t0 TCP 172.17.0.12:52054->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 253u IPv4 338952960 0t0 TCP 172.17.0.12:52328->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 254u IPv4 338952901 0t0 TCP 172.17.0.12:38980->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 255u IPv4 338960641 0t0 TCP 172.17.0.12:54454->10.95.112.43:2376 (CLOSE_WAIT)
python 7 164 root 256u IPv4 338952924 0t0 TCP 172.17.0.12:39000->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 258u IPv4 338960798 0t0 TCP 172.17.0.12:41390->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 259u IPv4 338961942 0t0 TCP 172.17.0.12:41426->10.95.112.42:2376 (CLOSE_WAIT)
python 7 164 root 260u IPv4 338961745 0t0 TCP 172.17.0.12:54628->10.95.112.43:2376 (CLOSE_WAIT)
[root@403eeb033b19 /]#
[root@403eeb033b19 /]# lsof -n -P | grep TCP | grep 2376 | grep CLOSE | wc -l
16632

Start with the session_scope context manager and the repeated prune_volumes calls shown in the report. Inspect how the Docker SDK closes TLS connections after these operations and reproduce the workload while monitoring lsof for CLOSE_WAIT sockets. Done means repeated operations no longer leave an unbounded number of connections in CLOSE_WAIT.