[Feature] Add distinct parameter support to FieldListaggAgg operator
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Users often store complex data such as `user1, user2, user3` as simple string rather then complex date type (map,array).
When aggregating these fields, there is currently no support for deduplication `set.add`.
### Solution
Add a `distinct` option support to FieldListAgg?
### Anything else?
This approach works fine for `user1,user2,user3`, but it's insufficient for kv-string deduplication `k1=v1;k2=v2`. I'm not sure whether to handle this with string manipulation or a map-merge aggregation approach due to performance concerns..
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The entry point named by the issue is the FieldListaggAgg operator; start by locating its implementation and related aggregation tests. Determine how a distinct option should apply to comma-separated values and clarify whether key-value string deduplication is in scope. Done means the supported distinct behavior is implemented and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100