apache / apache/fluss

ReplicaManager caches the projectionInfo to reduce the resource consumption for building projections.

Open
#748 0 comments 0 reactions 1 assignee Claimed by @ljw-hit View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

When we tested the column pruning capability of the log table, we found that there were some CPU bottlenecks on the server side. The characteristic of the table is that it has an extremely large number of columns, reaching tens of thousands. The CPU on the server side was fully utilized. After analyzing the flame graph, we discovered that the consumption at the data processing level was not significant. The main issue was that there was a large consumption when comparing some data types during the construction of the projection. The major part of the CPU overhead on the Fluss server side occurs in the stage of constructing the projection in FileLogProjection, among which the equals operations of various data types account for nearly 65%.

The `LogFileProjection` doesn't implement caching, which means that each `fetch` request requires reconstructing it, resulting in high costs. To solve this problem, a projection cache should be built in the `ReplicaManager`.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.