apache / apache/paimon

[Feature] Aggregation function to a nested struct type

Open
#6,092 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

The only aggregate function currently supported for nested types is nestd_update.
There is no support for STRUCT type yet. For spark example
```sql
CREATE TABLE t(
k INT,
a STRUCT,
g INT
)
TBLPROPERTIES ('primary-key' = 'k', 'merge-engine' = 'partial-update',
'fields.g.sequence-group'='a.a1,a.a2',
'fields.a.a1.aggregate-function'='sum',
'fields.a.a2.aggregate-function'='product'
)

INSERT INTO t
VALUES (1, STRUCT(1,3), 1);

INSERT INTO t
VALUES (1, STRUCT(2,4), 2);
```
The result expected is `1, STRUCT(3,12), 2`. However the actually output is `1, STRUCT(2,4), 2` because of nested aggregate-function has not been implemented.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing nestd_update aggregate-function handling and the path that processes the Spark STRUCT example with fields.a.a1 and fields.a.a2 aggregation properties. Reproduce the two inserts from the issue and verify that the result becomes STRUCT(3,12) rather than STRUCT(2,4), while preserving the sequence and other fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.