apache / apache/datafusion

[datafusion-spark] Implement `map` function

Open
#20,357 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

neither datafusion-spark nor datafusion-comet expose the map expression implementation. so when you try to use map expression in spark sql, you will get:
```sql
Expected only Comet native operators, but found Project.
plan: Project [COMET: map is not supported]
datafusion hasnt had the map function, we should add one there before supporting it in comet.
```

https://github.com/apache/datafusion-comet/issues/3386

### Describe the solution you'd like

- Implement `ScalarUDFImpl` trait for new Map struct and expose it to datafusion-spark map module
- use `comparison_coercion` to automatically casting when returning type
- use `make_scalar_function`
- use the `map_from_keys_values_offsets_nulls` to unify spark specific logic and code reusing
- transform the concat flatted keys and values with reorder_indices and arrow take

### Describe alternatives you've considered

there are already existing map expression in datafusion nested, but to align with rules and reuse the code in the [spark/src/function/map/utils.rs](https://github.com/peterxcli/datafusion/blob/main/datafusion/spark/src/function/map/utils.rs), I still think it's better to create a new implementation for spark map expression

https://github.com/apache/datafusion/blob/6746007826ebd3fcb5614bf87183674435bbb134/datafusion/functions-nested/src/map.rs#L240-L301

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.