dask / dask/distributed

Serialization Error Messages Hidden

Open
#3,975 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am running a map_partitions that returns a dataframe with rows that have a large memory footprint. I dug out the following exception below from the error log.

At first glance, it doesn't really make sense because doing a simple test shows that dask can serialize dataframes:

>>> from distributed.protocol.serialize import serialize as s
>>> s(pd.DataFrame())

However, taking a closer look, it appears that there is a generic Exception catch here:
https://github.com/dask/distributed/blob/2.21.0/distributed/protocol/serialize.py#L230

and if the on_error is set to raise like in the case of of moving partitions to disk during a map_partitions computation, we never get to see the message that is in the traceback format exception as shown here:

https://github.com/dask/distributed/blob/2.21.0/distributed/protocol/serialize.py#L234-L244

I suspect I am actually getting some sort of memory error, although I really have no way of knowing because the exception is reported as a TypeError

> 2020-07-21T17:12:03.599113035Z Traceback (most recent call last):
> 2020-07-21T17:12:03.599138316Z File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
> 2020-07-21T17:12:03.599143053Z ret = callback()
> 2020-07-21T17:12:03.599146719Z File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
> 2020-07-21T17:12:03.599150027Z future.result()
> 2020-07-21T17:12:03.599153015Z File "/usr/local/lib/python3.6/site-packages/distributed/worker.py", line 2688, in memory_monitor
> 2020-07-21T17:12:03.599161122Z k, v, weight = self.data.fast.evict()
> 2020-07-21T17:12:03.599164768Z File "/usr/local/lib/python3.6/site-packages/zict/lru.py", line 89, in evict
> 2020-07-21T17:12:03.599192248Z cb(k, v)
> 2020-07-21T17:12:03.599196364Z File "/usr/local/lib/python3.6/site-packages/zict/buffer.py", line 60, in fast_to_slow
> 2020-07-21T17:12:03.599200149Z self.slow[key] = value
> 2020-07-21T17:12:03.599203704Z File "/usr/local/lib/python3.6/site-packages/zict/func.py", line 41, in __setitem__
> 2020-07-21T17:12:03.599207077Z self.d[key] = self.dump(value)
> 2020-07-21T17:12:03.599210971Z File "/usr/local/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 475, in serialize_bytelist
> 2020-07-21T17:12:03.599214257Z header, frames = serialize(x, **kwargs)
> 2020-07-21T17:12:03.599217334Z File "/usr/local/lib/python3.6/site-packages/distributed/protocol/serialize.py", line 244, in serialize
> 2020-07-21T17:12:03.59922069Z raise TypeError(msg, str(x)[:10000])
> 2020-07-21T17:12:03.599225052Z TypeError: ('Could not serialize object of type DataFrame.', ' ...
`

**Environment**:

- Dask version: 2.21.0
- Python version: Python 3.6.9
- Install method (conda, pip, source): pip
- OS: Linux 5792330308a0 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 GNU/Linux

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.