apache / apache/paimon

[Bug] Collect agg merge engine return duplicate list even we set filed distinct

Open
#4,999 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master
1.0
0.9
0.8

### Compute Engine

none

### Minimal reproduce step

1. create table with collect agg and distinct field

```
CREATE TABLE test_collect(
id INT PRIMARY KEY NOT ENFORCED,
f0 ARRAY
) WITH (
'merge-engine' = 'aggregation',
'fields.f0.aggregate-function' = 'collect',
'fields.f0.distinct' = 'true'
)
```

2. insert values

`INSERT INTO test_collect VALUES (1, ARRAY['A', 'B', 'A', 'A']);`

3. query the results.

`SELECT * FROM test_collect`

the result is as follows, it does not meet expectations
` 1, [A, B, A, A] `

### What doesn't meet your expectations?

Return deduplicate list value.
for above example, the result shoud be:

` 1, [A, B] `

### Anything else?

no

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the supplied SQL reproduction against the aggregation merge engine with fields.f0.distinct=true, then trace the collect aggregate's handling of the distinct setting. Done means the query returns [A, B] rather than [A, B, A, A] for the supplied input.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.