tiflash CN region cache enhancement in disaggregated mode
Open
Nobody has claimed this yet.
type/enhancement
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
Bug fix:
Enhancements:
- [major] hot region balance: In the tikv-cse architecture, the region size is set to 1G instead of 96M. This means that many MPPTasks, which should have been distributed across multiple tiflash CNs, end up being assigned to just one CN. If the concurrency of such queries increases, it will lead to significantly high load on that single CN. Therefore, the task scheduling mechanism needs to support hotspot scheduling in addition to consistent hashing.
- [minor] retry when dispatch EstablishDisaggTask: In the disaggregated mode, the
EstablishDisaggTaskrequest does not take retries into account during the scale-in or scale-out of the Worker Node (WN). This may result in requests being sent while a WN is scaling in, leading to RPC errors and ultimately query failures. A better approach would be to use backoff and retry mechanisms. - [minor] detect new region info actively: If a region initially has only one TiFlash replica, the RegionCache will cache this information (e.g., caching
<region-1, <peer-1 on store-1>>). Later, if a new TiFlash replica is added to the region, making it<region-1, <peer-1 on store-1>, <peer-2 on store-2>>, the RegionCache may still not be aware of the new peer-2 because it only refreshes when the region becomes invalid (e.g., the corresponding store goes offline). As a result, the newly added peer-2 will not be recognized by the RegionCache, which can prevent the new replica from being utilized, leading to poor load balancing. - [minor] detect stale region cache actively: Better to add a background thread(or ttl) to update region cache for region cache. This can help alleviate the issue where a customer's first query, when encountering WN scale-in, always needs to perform a retry during batchCopTask construction in order to obtain the latest region information.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing RegionCache, MPPTask scheduling, and EstablishDisaggTask paths in TiFlash's disaggregated mode. Break the issue into the four listed enhancements—hot-region scheduling, dispatch retry, active new-region detection, and stale-cache refresh—and define validation for each before implementation. Done requires agreed scope and verified behavior for each enhancement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100