Consolidate map lookup code between `get_field` and `map_extract`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
DataFusion has two SQL constructs to lookup a key `k` in a map `m`, `m[k]` (planned as `get_field`) and `map_extract(m, k)`
Their APIs are slightly different (`get_field` returns the value or NULL, `map_extract` returns a singleton list or an empty list), but the core work is the same. There are three copies of the underlying key lookup code with different bugs and performance characteristics:
1. `map_extract` (after #24999): one arrow-ord comparator over the whole batch, per-row lookup keys, early exit per row, output sized to the number of matches.
2. `get_field::process_map_array`: a vectorized eq kernel between the scalar key and every map key, then a scan of the resulting bitmap. We take this path incorrectly for maps with list-typed keys, resulting in a panic (#25082)
3. `get_field::process_map_with_nested_key`: a comparator that compares each entry against the lookup key.
Since the `eq` kernel doesn't handle nested lists/maps, we probably want two code paths, but either way we can simplify and unify the code over the current state.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.