dask / dask/distributed

Closing dangling stream

Open
#2,507 13 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm creating temporary Clients to run custom task graphs on my remote cluster and was having problems with lots of socket connections sticking around and results stuck on the cluster due to exceptions causing the results on Futures to not be requested.

Explicitly calling Client.close() seemed to fix all that as I don't see any more stuck things on the cluster after an exceptions. But now I'm seeing a TCP connection not getting closed cleanly. Here's some code to reproduce the problem:

```
from dask.distributed import Client, as_completed
import uuid

def get_results():
c = Client('HOSTNAME:8786', set_as_default=False)

def daskfn():
return 'results'

futs = []
for i in range(100):
key = f'testfn-{uuid.uuid4()}'
futs.append(c.get({key: (daskfn, )}, key, sync=False))

results = []
try:
for f in as_completed(futs):
results.append(f.result())
finally:
c.close()
return results

results = get_results()
```

After this, using the command line program `ss`, I see a socket in CLOSE-WAIT state. And, if I do something that triggers garbage collection or call gc.collect(), I see the following warning:

```
distributed.comm.tcp - WARNING - Closing dangling stream in
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.