apache / apache/amoro

[Feature]: Support aggregation keys for mixed-format tables

Open
#1,359 1 comment 0 reactions 0 assignees View on GitHub
not-stale type:feature
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### Description

Data warehouses like Doris provide the ability to configure aggregation keys for tables. Data written to this field will be automatically aggregated before being written to the data file. This pre-computation capability greatly improves the analytical performance of the table.

Arctic's Mixed-format design opens up rules for data merging, and we can extend an aggregation merge rule to handle such use cases.

### Use case/motivation

Support defining aggregation fields on tables, data written to this field will be automatically pre-aggregated according to the aggregation function to improve query performance.

### Describe the solution

Extend the merge rules for data merging and add aggregation merge rules.

### Subtasks

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

The issue identifies no files, tests, or entry points. Start by locating Arctic's existing mixed-format merge rules and review how merge behavior is configured. Done means supporting aggregation fields and aggregation merge rules so configured values are pre-aggregated before being written to data files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.