apache / apache/datafusion

Each Union branch should produce fields with `nullable: true` if any of the branches produces fields with `nullable: true` on the same position

Open
#15,394 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

It was found in #15242 that the nullability of `Literal` is simply based on its scalar value and does not take its nullability in the input schema into account.

### To Reproduce

This issue cannot be seen in the CLI, as it doesn't do any strict type checking of record batches when displaying results. However, it can be observed in SLTs where `normalize::convert_batches` ensures that each record batch's schema is the same as that of the first batch.

Take the following query for example:

```
select x, y, z from t3 union all by name select z, y, x, 'd' as zz from t3;
```

`UNION` is a case where record batch streams from each input will be merged into an output stream.

`zz` should be considered as nullable in the output schema because it is not found in the lhs. However, the current behavior is that record batches streamed from the rhs do not have `zz` as null because of the current implementation of `PhysicalExpr::nullable` for `Literal`.

### Expected behavior

The nullability of `Literal` should be determined by its surrounding context _and_ value, not only its value.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the PhysicalExpr::nullable implementation for Literal and the UNION schema-merging path described in the issue. Reproduce the query in an SLT and inspect normalize::convert_batches, then verify that the rhs batch's added zz field is nullable and schema normalization succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.