apache / apache/datafusion

Consolidate map lookup code between `get_field` and `map_extract`

Closed
#25,083 0 comments 0 reactions 1 assignee Claimed by @neilconway View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

DataFusion has two SQL constructs to lookup a key `k` in a map `m`, `m[k]` (planned as `get_field`) and `map_extract(m, k)`

Their APIs are slightly different (`get_field` returns the value or NULL, `map_extract` returns a singleton list or an empty list), but the core work is the same. There are three copies of the underlying key lookup code with different bugs and performance characteristics:

1. `map_extract` (after #24999): one arrow-ord comparator over the whole batch, per-row lookup keys, early exit per row, output sized to the number of matches.
2. `get_field::process_map_array`: a vectorized eq kernel between the scalar key and every map key, then a scan of the resulting bitmap. We take this path incorrectly for maps with list-typed keys, resulting in a panic (#25082)
3. `get_field::process_map_with_nested_key`: a comparator that compares each entry against the lookup key.

Since the `eq` kernel doesn't handle nested lists/maps, we probably want two code paths, but either way we can simplify and unify the code over the current state.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.