apache / apache/datafusion

deprecating `return_type` in favor of `return_fields_from_args`

Open
#15,662 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Having `return_type` does not really reflect the UDF actual nullability.

the following problems exists when UDF wrongly report their nullability:

## Disallow optimization based on the expressions nullability.

For example lets assume I have 2 columns:
1. `col1` which is i64 without nulls
2. `col2` which is i64 with nulls

and the expression `ceil` does not does not implement `return_fields_from_args`, so by default the `nullability` is `true`

can't optimize the following query and

this means that I can can't optimize this query
```
select coalesce(ceil('col1'), 'col2') from tbl;
```

to be this:
```
select ceil('col1') from tbl;
```

because `ceil` says that the nullability is true even though the input expression is not nullable

the same can be for not able to remove the expression `array_remove_all(array, null)` if the array expression report that it is nullable while it is not

## Wrong types

creating the wrong type:

```
> select arrow_typeof(make_array(1));
+------------------------------------------------------------------------------------------------------------------+
| arrow_typeof(make_array(Int64(1))) |
+------------------------------------------------------------------------------------------------------------------+
| List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) |
+------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.002 seconds.
```

should be nullable `false` but because `make_array` implement `return_type` it does not get that information

Contributor guide

Open the contributing guide

Research direction

Start by locating the UDF implementations that use `return_type`, especially `ceil` and `make_array`, and compare them with `return_fields_from_args`. Trace the nullability and type inference tests, then confirm that affected UDFs derive those fields from their arguments and that the existing SQL examples produce the expected results.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend-api-design, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.