pingcap / pingcap/tiflash

tiflash CN region cache enhancement in disaggregated mode

Open
#10,203 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

Bug fix:

Enhancements:

  • [major] hot region balance: In the tikv-cse architecture, the region size is set to 1G instead of 96M. This means that many MPPTasks, which should have been distributed across multiple tiflash CNs, end up being assigned to just one CN. If the concurrency of such queries increases, it will lead to significantly high load on that single CN. Therefore, the task scheduling mechanism needs to support hotspot scheduling in addition to consistent hashing.
  • [minor] retry when dispatch EstablishDisaggTask: In the disaggregated mode, the EstablishDisaggTask request does not take retries into account during the scale-in or scale-out of the Worker Node (WN). This may result in requests being sent while a WN is scaling in, leading to RPC errors and ultimately query failures. A better approach would be to use backoff and retry mechanisms.
  • [minor] detect new region info actively: If a region initially has only one TiFlash replica, the RegionCache will cache this information (e.g., caching <region-1, <peer-1 on store-1>>). Later, if a new TiFlash replica is added to the region, making it <region-1, <peer-1 on store-1>, <peer-2 on store-2>>, the RegionCache may still not be aware of the new peer-2 because it only refreshes when the region becomes invalid (e.g., the corresponding store goes offline). As a result, the newly added peer-2 will not be recognized by the RegionCache, which can prevent the new replica from being utilized, leading to poor load balancing.
  • [minor] detect stale region cache actively: Better to add a background thread(or ttl) to update region cache for region cache. This can help alleviate the issue where a customer's first query, when encountering WN scale-in, always needs to perform a retry during batchCopTask construction in order to obtain the latest region information.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing RegionCache, MPPTask scheduling, and EstablishDisaggTask paths in TiFlash's disaggregated mode. Break the issue into the four listed enhancements—hot-region scheduling, dispatch retry, active new-region detection, and stale-cache refresh—and define validation for each before implementation. Done requires agreed scope and verified behavior for each enhancement.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.