dask / dask/dask

dask.dataframe.read_*: change default blocksize to 128 MiB

Open
#9,850 1 comment 0 reactions 0 assignees View on GitHub
dataframe enhancement good second issue io
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

- Related: https://github.com/dask/dask/issues/9849.

Oversized chunks cause the RAM-per-host requirements of a workload to shoot up.
High variance in the chunk size of a dataset confuses the duration prediction algorithm in the distributed scheduler - and probably a few other heuristics that I can't think of right now.

It is exceedingly common in real life to have to deal with loading files from disk that are far too large for comfort and/or vary wildly in size among the same dataset. I had recently to load up a dataset where the files varied between 22 MiB and 830 MiB each. Once decompressed, the 830 MiB partitions took in excees of 6 GiB RAM each to load up.

New dask users are unlikely to be immediately aware of this problem, which will cause their computatio to crash, and they may not notice that there's a simple parameter that would fix their problem.
*Experienced* dask uses are also not guaranteed to notice that there are a dozen of huge files among a dataset of 1000+ modestly sized ones.

# Proposed design
In an effort to improve user experience, I believe we should change the default `blocksize` for all `dask.dataframe.read_*` functions from False (1 file = 1 partition) to 128 MiB, coherently with the default chunk size in dask.array.
Note that 128 MiB is the size on disk, which can be substantially different from the size in memory. However I believe that ending up with somewhat too large or too small, but constrained, partitions is much preferable to having to deal with unconstrained partitions.

### Special cases

- `read_parquet` and `read_json` (for non-line json) don't support `blocksize`. It should be added (#9849).
- For the two above file formats, print a warning:
> Some partitions on disk are up to 830 MiB in size; they will be split into 128 MiB ones but only after loading them up at once. This will cause your RAM usage to spike during data load. If you produced this dataset yourself, consider saving it in smaller partitions (e.g. by calling `repartition` just before writing to disk). Explicitly set the blocksize parameter to disable this warning.

(only print the warning if there are partitions exceeding 256 MiB).

- `read_hdf` has `chunksize` (number of rows) instead of `blocksize` (number of bytes). We should add a mutually-exclusive `blocksize` parameter, which would crudely estimate `size of 1 row in bytes = size of the file in bytes / number of rows in the file`. The default when neither is explicitly stated should change from `chunksize=1_000_000` to `blocksize="128MiB"`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.