apache / apache/datafusion

UDTF arguments preserving

Open
#20,293 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Type coercion and simplification applied to UDTF arguments may break some functions with specific arguments parsing logic. e.g. in our project we support UDTF that allows to specify index to scan a table: `scan_with(t, index=[ta, tb])`. Using `SessionContextProvider` as `ContextProvider` the next error is returned:

```
Schema error: No field named index.
```

It caused by the type coercion here:

https://github.com/apache/datafusion/blob/4e2c0f1feaef2452323ada9c394ad943eb0b2805/datafusion/core/src/execution/session_state.rs#L1845-L1852

It would be nice to allow UDTF itself to handle arguments parsing. We could add a flag that will customize this behavior, allowing to pass args as is.

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/execution/session_state.rs at the type-coercion code around lines 1845-1852, and review how UDTF arguments reach that point. Reproduce the reported scan_with(t, index=[ta, tb]) case if possible. Done means UDTFs can opt to receive arguments without the coercion or simplification that currently produces the missing-field error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.