apache / apache/datafusion

`map[key]` misbehaves for NULL map rows that have entries

Open
#25,210 0 comments 0 reactions 1 assignee Claimed by @neilconway View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`map[key]` decides which entries belong to a row by considering only the map's offsets, not the validity buffer. However, the Arrow spec allows an element of a map or list to be marked invalid/NULL in the validity buffer but still to have a non-empty offset range (such an element should be considered NULL).

One way to get this in practice is via `nullif`, which is implemented by reusing the input's offsets/entries with an updated validity buffer.

### To Reproduce

```sql
CREATE TABLE t AS
SELECT MAP {'a': 1, 'b': 2} AS m, MAP {'a': 1, 'b': 2} AS n
UNION ALL
SELECT MAP {'a': 3, 'b': 4}, MAP {'x': 9};

SELECT NULLIF(m, n) AS nulled, NULLIF(m, n)['b'] AS value FROM t;
```

Produces

```
+--------------+-------+
| nulled | value |
+--------------+-------+
| NULL | 2 |
| {a: 3, b: 4} | 4 |
+--------------+-------+
```

### Expected behavior

```
+--------------+-------+
| nulled | value |
+--------------+-------+
| NULL | NULL |
| {a: 3, b: 4} | 4 |
+--------------+-------+
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.