apache / apache/datafusion

EPIC: Tuning guide for DataFusion

Open
#16,178 3 comments 0 reactions 0 assignees View on GitHub
enhancement EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion currently has around 100 configuration settings: https://datafusion.apache.org/user-guide/configs.html — and the number is still growing.

It can be tricky to tune configurations based solely on the brief descriptions in the documentation. Some settings require more context to fully understand their semantics.

A better approach would be to group related configurations and explain them with examples — effectively creating a tuning guide.

We could place the main content for this guide into separate examples under the `datafusion-examples` repository. This way, we can provide executable code alongside the documentation, and link to these examples from a dedicated doc page.

Here are two configuration groups I think would particularly benefit from such a guide:
- Join-related configurations https://github.com/apache/datafusion/issues/16176
- Configurations for out-of-core (larger-than-memory) execution https://github.com/apache/datafusion/issues/16177
- ... (suggestions for additional topics are welcome!)

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

DataFusion Comet has a great tuning guide at https://datafusion.apache.org/comet/user-guide/tuning.html

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing configuration documentation at datafusion.apache.org/user-guide/configs.html and the related join and out-of-core issues. Review the proposed datafusion-examples repository approach and the linked Comet tuning guide for context. Done means a dedicated tuning guide with grouped configuration explanations, executable examples, and links to those examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.