[datafusion-spark] Implement `map` function
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
neither datafusion-spark nor datafusion-comet expose the map expression implementation. so when you try to use map expression in spark sql, you will get:
```sql
Expected only Comet native operators, but found Project.
plan: Project [COMET: map is not supported]
datafusion hasnt had the map function, we should add one there before supporting it in comet.
```
https://github.com/apache/datafusion-comet/issues/3386
### Describe the solution you'd like
- Implement `ScalarUDFImpl` trait for new Map struct and expose it to datafusion-spark map module
- use `comparison_coercion` to automatically casting when returning type
- use `make_scalar_function`
- use the `map_from_keys_values_offsets_nulls` to unify spark specific logic and code reusing
- transform the concat flatted keys and values with reorder_indices and arrow take
### Describe alternatives you've considered
there are already existing map expression in datafusion nested, but to align with rules and reuse the code in the [spark/src/function/map/utils.rs](https://github.com/peterxcli/datafusion/blob/main/datafusion/spark/src/function/map/utils.rs), I still think it's better to create a new implementation for spark map expression
https://github.com/apache/datafusion/blob/6746007826ebd3fcb5614bf87183674435bbb134/datafusion/functions-nested/src/map.rs#L240-L301
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.