disagg: EstablishDisaggTask prone to timeout when the query involve a large amount of data

Enhancement

• 2 TiDB server
• 8 TiKV server
• 1 PD server
• 1 TiFlash WN
• 1 TiFlash CN

When executing select count(*) from tbl with about 237GB data in the delta layer, including 2GB delta-cache, EstablishDisaggTask prone to timeout and the query fails.

Now we serialize the delta-cache in the RPCEstablishDisaggTask, we'd better move it to the FetchPages streaming RPC

TiDB> select *,(TOTAL_SIZE/1024/1024/1024) as tot_gb,(TOTAL_DELTA_SIZE/1024/1024/1024) as tot_delta_gb,(DELTA_CACHE_SIZE/1024/1024/1024) as tot_delta_cache_gb from information_schema.tiflash_tables \G
***************************[ 1. row ]***************************
tot_gb                                    | 2895.740736620501
tot_delta_gb                              | 237.140938691795
tot_delta_cache_gb                        | 2.107738865539
TABLE_ID                                  | 102
SEGMENT_COUNT                             | 43332
TOTAL_ROWS                                | 1527761939
TOTAL_SIZE                                | 3109277940370
TOTAL_DELETE_RANGES                       | 0
DELTA_RATE_ROWS                           | 0.0817092027320102
DELTA_RATE_SEGMENTS                       | 0.25124619219052896
DELTA_PLACED_RATE                         | 0.020770056061652678
DELTA_CACHE_SIZE                          | 2263167374
DELTA_CACHE_RATE                          | 0.008888170769387164
DELTA_CACHE_WASTED_RATE                   | 0.0
DELTA_INDEX_SIZE                          | 86039336
AVG_SEGMENT_ROWS                          | 35257.12958091018
AVG_SEGMENT_SIZE                          | 71754775.69394442
DELTA_COUNT                               | 10887
TOTAL_DELTA_ROWS                          | 124832210
TOTAL_DELTA_SIZE                          | 254628144056
AVG_DELTA_ROWS                            | 11466.171580784421
AVG_DELTA_SIZE                            | 23388274.46091669
AVG_DELTA_DELETE_RANGES                   | 0.0
STABLE_COUNT                              | 25092
TOTAL_STABLE_ROWS                         | 1402929729
TOTAL_STABLE_SIZE                         | 2854649796314
TOTAL_STABLE_SIZE_ON_DISK                 | 783472671135
AVG_STABLE_ROWS                           | 55911.43507890961
AVG_STABLE_SIZE                           | 113767328.08520643
TOTAL_PACK_COUNT_IN_DELTA                 | 160301
MAX_PACK_COUNT_IN_DELTA                   | 20
AVG_PACK_COUNT_IN_DELTA                   | 14.724074584366676
AVG_PACK_ROWS_IN_DELTA                    | 778.7363148077679
AVG_PACK_SIZE_IN_DELTA                    | 1588437.6520171429
TOTAL_PACK_COUNT_IN_STABLE                | 190446
AVG_PACK_COUNT_IN_STABLE                  | 7.589909134385461
AVG_PACK_ROWS_IN_STABLE                   | 7366.548675215022
AVG_PACK_SIZE_IN_STABLE                   | 14989287.232674878

Start by tracing the EstablishDisaggTask RPC and the FetchPages streaming RPC in the TiFlash codebase, focusing on where the delta-cache is serialized. Reproduce the large-delta-layer count query if possible, then verify that delta-cache transfer no longer causes EstablishDisaggTask to time out and is handled through FetchPages.