Deduplicate Spark function code with native/default datafusion function code
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Some Spark functions have equivalent DataFusion functions, differing slightly in behaviour. Work towards reducing duplicated code, calling out any differences between the functions when we can't merge them together.
Functions with Spark on left, DF (closest) equivalent on right:
- [ ] length -> character_length
- [x] like -> like https://github.com/apache/datafusion/pull/19324
- Not much to deduplicate
- [ ] ilike -> ilike
- [ ] char -> chr
- [x] ascii -> ascii
- Done by #17965, though we could explore further deduplication
- [ ] rint -> round (?)
- [ ] mod/pmod -> modulo (?)
- [ ] hex -> to_hex (?)
- [ ] factorial -> factorial
- [x] ~~expm1 -> exp~~
- Won't do, see https://github.com/apache/datafusion/issues/17964#issuecomment-3496758810
- [ ] map_from_arrays -> map (?)
- [ ] sha1/sha2 -> sha* - @Jefffrey planning to look into this
- [ ] avg -> avg
- [x] array -> make_array https://github.com/apache/datafusion/pull/19006
Not all of the above are exact equivalents, they're just listed to make it easier to check if we can reduce duplication to some degree. I've listed the Spark functions I noticed that seem to have something similar in DataFusion already.
Contributor guide
Research direction
Start by reviewing the checked and unchecked Spark/DataFusion function pairs listed in the issue, along with the linked pull requests and discussion. Compare each pair's behavior before deciding whether duplication can be reduced. Done means applicable pairs are deduplicated and remaining differences are explicitly called out in the checklist or discussion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100