apache / apache/datafusion

Library Guide: Extending DataFusion's operators: custom LogicalPlan and `ExecutionPlans`

Open
#7,308 4 comments 2 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Part of https://github.com/apache/arrow-datafusion/issues/7014

If we want to have DataFusion used as the core of many new systems, we need it to be as easy as possible for someone to get their idea working on top of DataFusion.

Thanks to @tshauck we now have a basic Library Users Guide ❤️ and this ticket describes expanding it out

### Describe the solution you'd like

Fill in the content of https://arrow.apache.org/datafusion/library-user-guide/extending-operators.html

We can draw inspiration from https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/user_defined/user_defined_plan.rs

Example Outline
1. Introduce an example plan node that can not be expressed with existing relational operators (maybe pivot rows to columns, [like here](https://github.com/influxdata/influxdb_iox/blob/main/iox_query/src/exec/schema_pivot.rs))
2. Show how to define the Logical extension user defined node
3. SHow how to use an extension planner physical planner to plan such a node ([example here](https://github.com/influxdata/influxdb_iox/blob/1cc0926a7f52c7138be991df6dd60d11c6549df6/iox_query/src/exec/context.rs#L69-L85))
4. Show how to create a simplified execution plan / stream

The examples directory holds a bunch more of examples: https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with arrow.apache.org/datafusion/library-user-guide/extending-operators.html and the existing example in datafusion/core/tests/user_defined/user_defined_plan.rs. Review related examples in datafusion-examples and the linked InfluxDB IOX planning examples. Done means the guide explains a custom logical node, extension physical planner, and simplified execution plan or stream with working examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.