apache / apache/datafusion

Blog post about processing larger than memory queries

Open
#17,089 6 comments 2 reactions 1 assignee Claimed by @kiran-4444 View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

- part of https://github.com/apache/datafusion/issues/14836

As part of trying to help people understand how to tune memory related settings in https://github.com/apache/datafusion/pull/17069, @2010YOUY01 pointed out https://github.com/apache/datafusion/pull/17069#issuecomment-3167062577

> Let's make it concise now, I think adding a few more sentences of explanation might actually confuse those without the background knowledge. A tutorial-style doc is still needed to describe the full picture.

I agree and I think a blog post explaining how DataFusion processes larger than memory data sets would be really helpful to give people the broader context

### Describe the solution you'd like

I suggest a blog on https://datafusion.apache.org/blog/ (make a PR to https://github.com/alamb/datafusion-site)

### Describe alternatives you've considered

Some ideas

1. **Background** on memory usage in DataFusion (the memory manager model and what takes large amounts of memory) -- can probably copy/paste from https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#memory-management-overview
2. **Memory Consumers** Describe at a high level what consumes most memory in a plan (group by hash table, sorting, etc)
3. **Optimizations** that DataFusion does to try and avoid needing memory (e.g. take advantage of pre-xisting sort orders, topk, etc)
4. **Spilling Sort**: Provide an overview of the main spilling sort algorithm (sort in memory, spill to disk, merge pre-sorted runs, etc)
5. **Using Spilling Sort in Group By**: Explain that the grouping operation uses the same underlying building block
6. **Call for help**: 🎣 for people to figure out how to add spilling for joins and make the spilling hash faster

### Additional context

I am very happy to help write this

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.