Thoughts on additional spilling layers
Open
discussion
memory
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Have had a few thoughts about additional spilling layers we could benefit from:
* Defragmentation (perform a `memcpy` into larger and larger buffers to clean up and consolidate small buffers)
* Shared memory (also has the benefit of being accessible from other workers on the same node)
* mmap files (in-between shared memory and a file on disk; can close when memory usage is high; also like memory when using solid state)
Would be straightforward to add these to Zict without any new dependencies
Contributor guide
Assessment
This issue has not been assessed yet.