UDTF arguments preserving
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Type coercion and simplification applied to UDTF arguments may break some functions with specific arguments parsing logic. e.g. in our project we support UDTF that allows to specify index to scan a table: `scan_with(t, index=[ta, tb])`. Using `SessionContextProvider` as `ContextProvider` the next error is returned:
```
Schema error: No field named index.
```
It caused by the type coercion here:
https://github.com/apache/datafusion/blob/4e2c0f1feaef2452323ada9c394ad943eb0b2805/datafusion/core/src/execution/session_state.rs#L1845-L1852
It would be nice to allow UDTF itself to handle arguments parsing. We could add a flag that will customize this behavior, allowing to pass args as is.
Contributor guide
Research direction
Start in datafusion/core/src/execution/session_state.rs at the type-coercion code around lines 1845-1852, and review how UDTF arguments reach that point. Reproduce the reported scan_with(t, index=[ta, tb]) case if possible. Done means UDTFs can opt to receive arguments without the coercion or simplification that currently produces the missing-field error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100