apache / apache/datafusion

Scalars are too verbose in column name output

Open
#15,395 10 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 11h
Merged PRs (30d)
362

Description

### Is your feature request related to a problem or challenge?

When parsing scalar expressions, DataFusion makes queries quite complicated:

```sql
> select 3, array_length([1, 2, 4, 5, 10000, 2.4]);;
+----------+-----------------------------------------------------------------------------------------+
| Int64(3) | array_length(make_array(Int64(1),Int64(2),Int64(4),Int64(5),Int64(10000),Float64(2.4))) |
+----------+-----------------------------------------------------------------------------------------+
| 3 | 6 |
+----------+-----------------------------------------------------------------------------------------+
```

Such detailed comparison isn't needed in most cases and adds unnecessary cognitive complexity for developers. Moreover, it also breaks the original query - if you put `select Int64(3)`, you'll get `Error during planning: Invalid function 'int64'`

### Describe the solution you'd like

In most cases, the simplest version will be enough. On the example above it is:

```sql
> select 3, array_length([1, 2, 4, 5, 10000, 2.4]);
+---+---------------------------------------------+
| 3 | array_length(make_array(1,2,4,5,10000,2.4)) |
+---+---------------------------------------------+
| 3 | 6 |
+---+---------------------------------------------+
```

IMO there are three ways how to fix this problem:

1. Just remove `:?` here for all cases
https://github.com/apache/datafusion/blob/0ff89844a2f2c7a3bbcaff995db7e464deaeb997/datafusion/expr/src/expr.rs#L2951
This will make queries simpler everywhere. The downside is that we lose some information, e.g., type info in the shell (which one can argue isn't needed and can be checked separately)
2. Disable verbose as a `ConfigOptions` param and make it changeable (via datafusion-cli or sdk).
3. Use short names only if parsing back preserves the correct type. For example, when formatting `Int64(0)`, output `0` (since parsing `0` results in `Int64(0)`). When formatting `Int32(0)`, keep `Int32(0)`.

### Additional context

May be quite easy to update after we finish https://github.com/apache/datafusion/issues/15178

Contributor guide

Open the contributing guide

Research direction

Start with the scalar expression formatting code at datafusion/expr/src/expr.rs around line 2951, then read issue #15178 for the stated dependency. Decide which of the three formatting approaches preserves the needed type information, update the column-name output accordingly, and verify that the displayed SQL no longer uses unnecessarily verbose scalar names.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.