dask / dask/distributed

Compress larger buffers

Open
#6,286 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently we disallow compressing buffers of a certain size

https://github.com/dask/distributed/blob/7bd64425225af4847e2fbf045df31fb60a253e7d/distributed/protocol/compression.py#L161

This tracks back to issue ( https://github.com/dask/distributed/issues/366 ) and PR ( https://github.com/dask/distributed/pull/367 ). AIUI this was added to workaround a Blosc issue, which is no longer supported ( https://github.com/dask/distributed/pull/6027 ). Though LZ4 has a similar issue as was discovered in Numcodecs ( https://github.com/zarr-developers/numcodecs/pull/81 ).

As noted in comment ( https://github.com/dask/distributed/pull/6273#discussion_r866246847 ), this may be due to the use of `int32` for buffer sizes in compression algorithms. Not entirely sure why that is. Though it could be a technical or practical limitation (2GB is a pretty big buffer).

It might be worth investigating whether compressors still have this limitation and if so how we want to handle it. For example if it still exists, we could break large buffers up and compress smaller chunks to workaround this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.