apache / apache/datafusion

[EPIC] Complete `datafusion-spark` Spark Compatible Functions

Open
#15,914 29 comments 14 reactions 0 assignees View on GitHub
enhancement EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Many DataFusion users are using DataFusion to execution workloads originally developed for Apache Spark. Examples include
- [DataFusion Comet](https://datafusion.apache.org/comet/) (@andygrove @comphead , etc)
- [LakeHQ / Sail](https://github.com/lakehq/sail) (@shehabgamin )
- Various internal pileines / engines (e.g. that @Omega359 and I think @Blizzara use)

They often do this for superior performance

* Part of running Spark workloads is emulating Spark sematics
* Emulating Spark semantics requires (among other things) functions compatible with Spark (which differs in semantics to the functions included in DataFusion)

Several projects are in the process of implementing Spark compatible function libraries using DataFusion's extension APIs. However. we concluded in https://github.com/apache/datafusion/issues/5600 that we could join forces and maintain a spark compatible funciton library in the core datafusion repo. @shehabgamin has implemented the first step in https://github.com/apache/datafusion/pull/15168 🙏

### Describe the solution you'd like

This ticket tracks "completing" the spark function library started in https://github.com/apache/datafusion/pull/15168

### Describe alternatives you've considered

`datetime` functions:
- [x] https://github.com/apache/datafusion/issues/16774
- [x] https://github.com/apache/datafusion/issues/16775

`string` functions:
- [x] https://github.com/apache/datafusion/issues/16612

`math` functions:
- [x] https://github.com/apache/datafusion/issues/15916

`set` functions:
- [x] https://github.com/apache/datafusion/issues/17924
- [x] https://github.com/apache/datafusion/issues/17923

`map`/`array` functions:
- [x] https://github.com/apache/datafusion/issues/5338

Infrastructure and Testing:
- [x] https://github.com/apache/datafusion/issues/5600
- [x] https://github.com/apache/datafusion/pull/15168
- [x] https://github.com/apache/datafusion/issues/15915
- [ ] https://github.com/apache/datafusion/issues/17045
- [x] https://github.com/apache/datafusion/issues/17987
- [ ] https://github.com/apache/datafusion/issues/17964

Related issues
- [ ] https://github.com/apache/datafusion-comet/issues/1704
- [ ] https://github.com/apache/datafusion-comet/issues/1819
- [ ] https://github.com/apache/datafusion/issues/11201
- [x] https://github.com/apache/datafusion/issues/16336
- [ ] https://github.com/apache/datafusion/pull/14367
- [x] https://github.com/apache/datafusion/issues/17455

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Spark-compatible function library introduced by pull request 15168 and review the linked checklist issues, especially the unchecked infrastructure and testing items 17045 and 17964. Use the completed function issues as context; this epic is done when the remaining Spark-compatible functions and listed infrastructure work are completed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.