Proper support for Array/List type
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Pinot currently has a multi-value column type: which essentially is a ordered, bag-of-elements, set data structure.
However, many requirement comes in to ask for proper array support (see https://github.com/apache/pinot/issues/6083)
This proposal is for proper array/list type as a separate DataType.
- [x] conversion between MV and Array/List type should be supported (`CAST`)
- NOTE: as @Jackie-Jiang mentioned, we can directly operate on MV columns since they are ordered arrays
- [ ] array operations should be done on array type, including
- [ ] predicates (`match_all`, `match_none`, `match_any` or `contains`)
- https://github.com/apache/pinot/pull/9323
- [x] whole array operations producing a single value (`sum`/`min`/`max`/`cardinality`)
- cardinality supported in: https://github.com/apache/pinot/pull/9300; sum/min/max already supported in https://github.com/apache/pinot/pull/6084
- [ ] element-wise operations (such as `filter`, `transform`)
- [ ] indexed access (`arr[1]` or `element_at`)
Contributor guide
Research direction
Start with Pinot's DataType handling and the linked issue and pull requests referenced in this proposal. Review the remaining unchecked work for array predicates, element-wise operations, and indexed access, along with the completed CAST and whole-array operations. Done would require an agreed implementation scope and support for the remaining array/list behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100