EPIC: Tuning guide for DataFusion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion currently has around 100 configuration settings: https://datafusion.apache.org/user-guide/configs.html — and the number is still growing.
It can be tricky to tune configurations based solely on the brief descriptions in the documentation. Some settings require more context to fully understand their semantics.
A better approach would be to group related configurations and explain them with examples — effectively creating a tuning guide.
We could place the main content for this guide into separate examples under the `datafusion-examples` repository. This way, we can provide executable code alongside the documentation, and link to these examples from a dedicated doc page.
Here are two configuration groups I think would particularly benefit from such a guide:
- Join-related configurations https://github.com/apache/datafusion/issues/16176
- Configurations for out-of-core (larger-than-memory) execution https://github.com/apache/datafusion/issues/16177
- ... (suggestions for additional topics are welcome!)
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
DataFusion Comet has a great tuning guide at https://datafusion.apache.org/comet/user-guide/tuning.html
Contributor guide
Research direction
Start by reading the existing configuration documentation at datafusion.apache.org/user-guide/configs.html and the related join and out-of-core issues. Review the proposed datafusion-examples repository approach and the linked Comet tuning guide for context. Done means a dedicated tuning guide with grouped configuration explanations, executable examples, and links to those examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100