apache / apache/paimon

[Feature] Add distinct parameter support to FieldListaggAgg operator

Open
#6,567 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Users often store complex data such as `user1, user2, user3` as simple string rather then complex date type (map,array).
When aggregating these fields, there is currently no support for deduplication `set.add`.

### Solution

Add a `distinct` option support to FieldListAgg?

### Anything else?

This approach works fine for `user1,user2,user3`, but it's insufficient for kv-string deduplication `k1=v1;k2=v2`. I'm not sure whether to handle this with string manipulation or a map-merge aggregation approach due to performance concerns..

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The entry point named by the issue is the FieldListaggAgg operator; start by locating its implementation and related aggregation tests. Determine how a distinct option should apply to comma-separated values and clarify whether key-value string deduplication is in scope. Done means the supported distinct behavior is implemented and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.