[Feature]: Support aggregation keys for mixed-format tables
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 395
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 33
Description
### Description
Data warehouses like Doris provide the ability to configure aggregation keys for tables. Data written to this field will be automatically aggregated before being written to the data file. This pre-computation capability greatly improves the analytical performance of the table.
Arctic's Mixed-format design opens up rules for data merging, and we can extend an aggregation merge rule to handle such use cases.
### Use case/motivation
Support defining aggregation fields on tables, data written to this field will be automatically pre-aggregated according to the aggregation function to improve query performance.
### Describe the solution
Extend the merge rules for data merging and add aggregation merge rules.
### Subtasks
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
The issue identifies no files, tests, or entry points. Start by locating Arctic's existing mixed-format merge rules and review how merge behavior is configured. Done means supporting aggregation fields and aggregation merge rules so configured values are pre-aggregated before being written to data files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100