apache / apache/skywalking

[Feature] Implement batch TopN aggregation for historical data on non-hot nodes

Open
#13,286 0 comments 0 reactions 0 assignees View on GitHub
database feature
Dominant language
Java
Stars
25k
Forks
6.6k
Avg merge
10h 5m
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.

### Description

Related to #13285

Implement a batch processing system that enables TopN aggregation for historical data stored on warm and cold nodes. This feature will support scenarios where historical data is imported and users need to generate TopN aggregations retroactively with flexible configuration options.

### Background

Non-hot storage tiers (warm and cold nodes) often contain historical data that may be imported in bulk. Currently, there's no efficient mechanism to perform TopN aggregations on this historical data without impacting real-time operations or requiring manual intervention.

### Proposed Features

Develop a batch processing capability that supports:

- On-demand TopN aggregation through configurable batch tasks
- Flexible time-range specifications for historical data processing
- User-defined naming conventions for generated TopN results
- Support for importing historical data with subsequent batch aggregation processing

### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit a pull request to implement this on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.