JuliaParallel / JuliaParallel/DTables.jl

docs: Add some recommendations on table partitioning

Open
#40 0 comments 0 reactions 0 assignees View on GitHub
documentation good first issue
Dominant language
Julia
Stars
81
Forks
2
Avg merge
2h 31m
Merged PRs (30d)
1

Description

It's not entirely obvious to new users what value should be specified for `chunksize`, yet its choice is vital to getting good performance. It would be great if we could add to the docs some recommendations (and possibly examples) of how to choose a good `chunksize` value. Specifically, we could list a few basic recommendations:
- Specify it as a ratio of the total number of rows (e.g. `nrows / 10`)
- When the data is very big, limit it to a certain maximum size (e.g. whether data is 20GB or 300GB, pick `10_000_000` rows)
- If the same analysis will be rerun many times, benchmark different `chunksize` choices and pick the one that is fastest but also doesn't cause OOM errors
- etc.

Showing an example of how to do that benchmarking would be really useful to users who are lost on what `chunksize` to pick.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.