Add Support for Dynamic SQL Macros for Flexible Column Selection
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
People may need to write flexible SQL queries that adapt to different table schemas without hardcoding column names.
Query reusability is very limited currently, and creating dynamic queries requires external templating. A built-in mechanism to define and use dynamic SQL macros would significantly enhance the expressiveness and reusability
### Describe the solution you'd like
Introduce support for dynamic SQL macros that allow users to parameterize column selection. For example, a macro could accept lists of columns to include or exclude and then automatically select the appropriate columns from a given table or CTE. An example usage might look like this:
```sql
CREATE OR REPLACE MACRO dynamic_select(
include_cols,
exclude_cols
) AS TABLE (
FROM source_table
SELECT
COLUMNS(col ->
(list_contains(include_cols, col))
AND
(NOT list_contains(exclude_cols, col))
)
);
WITH source_table AS (
SELECT * FROM employees
)
FROM dynamic_select(
['id', 'name', 'department'],
['salary']
);
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the proposed CREATE OR REPLACE MACRO example, especially its COLUMNS expression, list_contains calls, and include/exclude parameters. Trace how DataFusion currently handles SQL macros and flexible column selection, then define tests for the employees-style query; done means dynamic macros can select included columns while excluding requested columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100