apache / apache/datafusion

[EPIC] Full lambda support

Open
#21,172 12 comments 4 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Some features haven't been implemented and others pulled out from ~#18921~ #21679 to keep it from getting too large (>5k LOC) and should be implemented in follow on PR's:

Major features:

- [x] #21323 to support construction like `select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end) `
- [x] #21193
- [x] #22101
- [x] #22326
- [ ] #22689
- [ ] #22853
- [ ] #22847
- [x] #22362
- [x] #22593 (also add alias support)
- [ ] add HigherOrderUDF to the [public user guide docs](https://datafusion.apache.org/library-user-guide/functions/adding-udfs.html)
- [x] #21895
- [ ] array_transform index parameter
- [x] #21903
- [x] #21896
- [x] #22107
- [x] #22106
- [x] #21950
- [x] #22097

Minor features:

- [ ] Named parameters like ScalarUDF already supports: `func(a => 3)`
- [ ] Expr simplifier optimizations for `LambdaVariable`
- [ ] ListView support in remove_list_null_values and HigherOrderUDF::clean_null_values
- [ ] Optional ScalarUDFImpl methods not present on HigherOrderUDF(some are unlikely to be used and could be added on demand only):
- [ ] simplify
- [ ] with_updated_config
- [ ] preimage
- [ ] placement
- [ ] evaluate_bounds
- [ ] propagate_constraints
- [ ] output_ordering
- [ ] preserves_lex_ordering

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the unchecked checklist items, especially HigherOrderUDF documentation, array_transform's index parameter, named parameters, and the listed ScalarUDFImpl methods. Read the existing HigherOrderUDF and ScalarUDF implementations before selecting one focused follow-on task; done means the selected feature is implemented and its checklist item is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.