[EPIC] Full lambda support
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Some features haven't been implemented and others pulled out from ~#18921~ #21679 to keep it from getting too large (>5k LOC) and should be implemented in follow on PR's:
Major features:
- [x] #21323 to support construction like `select a, exists(array(1, 2, 3), x-> case when a = 2 then x > 1 else null end) `
- [x] #21193
- [x] #22101
- [x] #22326
- [ ] #22689
- [ ] #22853
- [ ] #22847
- [x] #22362
- [x] #22593 (also add alias support)
- [ ] add HigherOrderUDF to the [public user guide docs](https://datafusion.apache.org/library-user-guide/functions/adding-udfs.html)
- [x] #21895
- [ ] array_transform index parameter
- [x] #21903
- [x] #21896
- [x] #22107
- [x] #22106
- [x] #21950
- [x] #22097
Minor features:
- [ ] Named parameters like ScalarUDF already supports: `func(a => 3)`
- [ ] Expr simplifier optimizations for `LambdaVariable`
- [ ] ListView support in remove_list_null_values and HigherOrderUDF::clean_null_values
- [ ] Optional ScalarUDFImpl methods not present on HigherOrderUDF(some are unlikely to be used and could be added on demand only):
- [ ] simplify
- [ ] with_updated_config
- [ ] preimage
- [ ] placement
- [ ] evaluate_bounds
- [ ] propagate_constraints
- [ ] output_ordering
- [ ] preserves_lex_ordering
Contributor guide
Research direction
Start by reviewing the unchecked checklist items, especially HigherOrderUDF documentation, array_transform's index parameter, named parameters, and the listed ScalarUDFImpl methods. Read the existing HigherOrderUDF and ScalarUDF implementations before selecting one focused follow-on task; done means the selected feature is implemented and its checklist item is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100