apache / apache/datafusion

Deduplicate Spark function code with native/default datafusion function code

Open
#17,964 27 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Some Spark functions have equivalent DataFusion functions, differing slightly in behaviour. Work towards reducing duplicated code, calling out any differences between the functions when we can't merge them together.

Functions with Spark on left, DF (closest) equivalent on right:

- [ ] length -> character_length
- [x] like -> like https://github.com/apache/datafusion/pull/19324
- Not much to deduplicate
- [ ] ilike -> ilike
- [ ] char -> chr
- [x] ascii -> ascii
- Done by #17965, though we could explore further deduplication
- [ ] rint -> round (?)
- [ ] mod/pmod -> modulo (?)
- [ ] hex -> to_hex (?)
- [ ] factorial -> factorial
- [x] ~~expm1 -> exp~~
- Won't do, see https://github.com/apache/datafusion/issues/17964#issuecomment-3496758810
- [ ] map_from_arrays -> map (?)
- [ ] sha1/sha2 -> sha* - @Jefffrey planning to look into this
- [ ] avg -> avg
- [x] array -> make_array https://github.com/apache/datafusion/pull/19006

Not all of the above are exact equivalents, they're just listed to make it easier to check if we can reduce duplication to some degree. I've listed the Spark functions I noticed that seem to have something similar in DataFusion already.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the checked and unchecked Spark/DataFusion function pairs listed in the issue, along with the linked pull requests and discussion. Compare each pair's behavior before deciding whether duplication can be reduced. Done means applicable pairs are deduplicated and remaining differences are explicitly called out in the checklist or discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, spark
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.