[Bug] Collect agg merge engine return duplicate list even we set filed distinct
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
1.0
0.9
0.8
### Compute Engine
none
### Minimal reproduce step
1. create table with collect agg and distinct field
```
CREATE TABLE test_collect(
id INT PRIMARY KEY NOT ENFORCED,
f0 ARRAY
) WITH (
'merge-engine' = 'aggregation',
'fields.f0.aggregate-function' = 'collect',
'fields.f0.distinct' = 'true'
)
```
2. insert values
`INSERT INTO test_collect VALUES (1, ARRAY['A', 'B', 'A', 'A']);`
3. query the results.
`SELECT * FROM test_collect`
the result is as follows, it does not meet expectations
` 1, [A, B, A, A] `
### What doesn't meet your expectations?
Return deduplicate list value.
for above example, the result shoud be:
` 1, [A, B] `
### Anything else?
no
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the supplied SQL reproduction against the aggregation merge engine with fields.f0.distinct=true, then trace the collect aggregate's handling of the distinct setting. Done means the query returns [A, B] rather than [A, B, A, A] for the supplied input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100