apache / apache/datafusion

Add Support for Dynamic SQL Macros for Flexible Column Selection

Open
#14,512 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

People may need to write flexible SQL queries that adapt to different table schemas without hardcoding column names.

Query reusability is very limited currently, and creating dynamic queries requires external templating. A built-in mechanism to define and use dynamic SQL macros would significantly enhance the expressiveness and reusability

### Describe the solution you'd like

Introduce support for dynamic SQL macros that allow users to parameterize column selection. For example, a macro could accept lists of columns to include or exclude and then automatically select the appropriate columns from a given table or CTE. An example usage might look like this:

```sql
CREATE OR REPLACE MACRO dynamic_select(
include_cols,
exclude_cols
) AS TABLE (
FROM source_table
SELECT
COLUMNS(col ->
(list_contains(include_cols, col))
AND
(NOT list_contains(exclude_cols, col))
)
);

WITH source_table AS (
SELECT * FROM employees
)
FROM dynamic_select(
['id', 'name', 'department'],
['salary']
);
```

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the proposed CREATE OR REPLACE MACRO example, especially its COLUMNS expression, list_contains calls, and include/exclude parameters. Trace how DataFusion currently handles SQL macros and flexible column selection, then define tests for the employees-style query; done means dynamic macros can select included columns while excluding requested columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.