dask / dask/distributed

Thoughts on additional spilling layers

Open
#4,629 5 comments 0 reactions 0 assignees View on GitHub
discussion memory
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Have had a few thoughts about additional spilling layers we could benefit from:

* Defragmentation (perform a `memcpy` into larger and larger buffers to clean up and consolidate small buffers)
* Shared memory (also has the benefit of being accessible from other workers on the same node)
* mmap files (in-between shared memory and a file on disk; can close when memory usage is high; also like memory when using solid state)

Would be straightforward to add these to Zict without any new dependencies

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.