apache / apache/datafusion

select multiple columns in a single `Expr`

Open
#10,102 22 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I want to create a udf that can select multiple functions at once, such as a `COLUMNS()` function.

`select COLUMNS('number\d+') from my_table`.

Currently this is not possible due to the fact that udfs can only ever output a single `Expr`.

### Describe the solution you'd like

Since it would be quite a massive overhaul to refactor all of the planning and udf logic to return `Vec`, I propose adding a new variant to `Expr`.

`Expr::Columns(Vec)`.

This seems like the least invasive way to support selecting multiple columns in a single expr.

### Describe alternatives you've considered

I'm open for alternatives, but I am not aware of any.

### Additional context

Polars has this variant in their `Expr`

https://github.com/pola-rs/polars/blob/9fec2ecb6d4295969e1d155b386ee82db08745a1/crates/polars-plan/src/dsl/expr.rs#L72

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Expr, UDF, and planning logic referenced in the issue, and compare the proposed Expr::Columns(Vec) with the linked Polars Expr variant. Define how one expression can select multiple columns without refactoring every planner and UDF return path; done means the requested COLUMNS() query can be represented and planned.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.