dask / dask/distributed

Add helpers to make scheduler state JSON serializable

Open
#4,126 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In a recent debugging session, I was trying to inspect a remote scheduler through a bunch of `client.run_on_scheduler(lambda dask_scheduler: dask_scheduler.)` commands. Some of these (like `.tasks`) were harder since they contain references to non-serializable objects.

WorkerState.identity() is I think what I have in mind. This would be similar for all the objects in the scheduler state, and would recursively apply to values in the result.

```pytb
>>> tasks = client.run_on_scheduler(lambda dask_scheduler: dask_scheduler.tasks)
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/protocol/core.py", line 151, in loads
value = _deserialize(head, fs, deserializers=deserializers)
File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 335, in deserialize
return loads(header, frames)
File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 71, in pickle_loads
return pickle.loads(x, buffers=buffers)
File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 75, in loads
return pickle.loads(x)
File "/srv/conda/envs/notebook/lib/python3.7/site-packages/distributed/scheduler.py", line 301, in __hash__
return hash(self.address)
AttributeError: address
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.