4paradigm / 4paradigm/OpenMLDB

Optimize concurrent access for aggregators

Open
#1,559 5 comments 0 reactions 2 assignees Claimed by @zhanghaohit View on GitHub
enhancement
Dominant language
C++
Stars
1.7k
Forks
331
Avg merge
12d 12h
Merged PRs (30d)
1

Description

**Describe the feature you'd like**

The aggregators may be accessed concurrently. Now each of the aggregators is protected by its mutex. It is costly in terms of memory usage.

We should design a more efficient thread-safe method to access the aggregators.

Some possible directions:

1. introduce the implementation of concurrent hashmap from third-party libraries.
2. asynchronous update for the aggregators
3. introduce some `atomic` way to eliminate the `lock`

**Additional context**
pre-aggregators update during `Put` request

src/storage/aggregator.h

Contributor guide

Open the contributing guide

Research direction

The aggregators are defined in src/storage/aggregator.h and are used during Put requests. Examine the current mutex usage and the concurrent access patterns. Research third-party concurrent hashmap libraries or atomic operations suitable for C++. A successful change will maintain thread safety while reducing memory overhead, and likely requires benchmarking.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.