apache / apache/datafusion

Support accessing a map with non-literal, non-constant key

Open
#14,552 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Accessing a map with a non-string key doesn't work (at least via DataFusion SQL) and is covered by https://github.com/apache/datafusion/issues/11785

Accessing a map with a literal string key works:

https://github.com/apache/datafusion/blob/5e1e693b6e30888d710aefc57be58ea25f62d34d/datafusion/sqllogictest/test_files/map.slt#L69-L72

Add support for accessing a map with non-literal key.
For example this should work (and also other cases where key cannot be constant-folded to a literal):

```
query ok
SELECT SUM(ints['by' || 'tes']) FROM data;
```

but it does not:
> `DataFusion error: Error during planning: Failed to coerce arguments to satisfy a call to 'array_element' function: coercion from [Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), Utf8] to the signature ArraySignature(ArrayAndIndexes(1)) failed No function matches the given name and argument types 'array_element(Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), Utf8)'. You might need to add explicit type casts.
Candidate functions:
array_element(array, index`

Contributor guide

Open the contributing guide

Research direction

The existing map case is in datafusion/sqllogictest/test_files/map.slt (L69-L72); start there and inspect the SQL planning path that produces the array_element coercion error. Add coverage for SELECT SUM(ints['by' || 'tes']) FROM data and confirm non-constant keys work without breaking literal-key access.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.