Background GC cause too many read/write amplification

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2025/07/31 06:43:16.104 +08:00] [INFO] [DeltaMergeStore_InternalBg.cpp:821] ["GC - Trigger MergeDelta, compact_reason=TooManyInvalidVersion segment=<segment_id=1 epoch=12025 range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=4294967295 table_id=1621"] [thread_id=132]
[2025/07/31 06:43:16.104 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:381] ["MergeDelta - Begin, reason=BackgroundGCThread safe_point=459755685378523136 segment=<segment_id=1 epoch=12025 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=26 delta_bytes=3900 delta_deletes=0 stable_file=dmf_5061747 stable_rows=9354 stable_bytes=1403100 dmf_rows=9354 dmf_bytes=1403100 dmf_packs=2>"] [source="keyspace=4294967295 table_id=1621"] [thread_id=132]
[2025/07/31 06:43:16.129 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:514] ["
MergeDelta - Finish, delta is merged, 
old_segment=<segment_id=1 epoch=12025 range=[-9223372036854775808,9223372036854775807) abandoned=true next_segment_id=0 delta_rows=26 delta_bytes=3900 delta_deletes=0 stable_file=dmf_5061747 stable_rows=9354 stable_bytes=1403100 dmf_rows=9354 dmf_bytes=1403100 dmf_packs=2> 
new_segment=<segment_id=1 epoch=12026 range=[-9223372036854775808,9223372036854775807)                next_segment_id=0 delta_rows=0 delta_bytes=0 delta_deletes=0 stable_file=dmf_5062091 stable_rows=9360 stable_bytes=1404000 dmf_rows=9360 dmf_bytes=1404000 dmf_packs=2>"] [source="keyspace=4294967295 table_id=1621"] [thread_id=132]

[2025/07/31 06:43:16.130 +08:00] [INFO] [DeltaMergeStore_InternalBg.cpp:821] ["GC - Trigger MergeDelta, compact_reason=TooManyInvalidVersion segment=<segment_id=1 epoch=21283 range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=4294967295 table_id=3294"] [thread_id=132]
[2025/07/31 06:43:16.130 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:381] ["MergeDelta - Begin, reason=BackgroundGCThread safe_point=459755685378523136 segment=<segment_id=1 epoch=21283 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=43 delta_bytes=6493 delta_deletes=0 stable_file=dmf_5061749 stable_rows=24011 stable_bytes=3625661 dmf_rows=24011 dmf_bytes=3625661 dmf_packs=3>"] [source="keyspace=4294967295 table_id=3294"] [thread_id=132]
[2025/07/31 06:43:16.165 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:514] ["MergeDelta - Finish, delta is merged, 
old_segment=<segment_id=1 epoch=21283 range=[-9223372036854775808,9223372036854775807) abandoned=true next_segment_id=0 delta_rows=43 delta_bytes=6493 delta_deletes=0 stable_file=dmf_5061749 stable_rows=24011 stable_bytes=3625661 dmf_rows=24011 dmf_bytes=3625661 dmf_packs=3> 
new_segment=<segment_id=1 epoch=21284 range=[-9223372036854775808,9223372036854775807)                next_segment_id=0 delta_rows=0 delta_bytes=0 delta_deletes=0 stable_file=dmf_5062092 stable_rows=24019 stable_bytes=3626869 dmf_rows=24019 dmf_bytes=3626869 dmf_packs=3>"] [source="keyspace=4294967295 table_id=3294"] [thread_id=132]

[2025/07/31 06:43:16.166 +08:00] [INFO] [DeltaMergeStore_InternalBg.cpp:821] ["GC - Trigger MergeDelta, compact_reason=TooManyInvalidVersion segment=<segment_id=1 epoch=20861 range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=4294967295 table_id=3409"] [thread_id=132]
[2025/07/31 06:43:16.166 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:381] ["MergeDelta - Begin, reason=BackgroundGCThread safe_point=459755685378523136 segment=<segment_id=1 epoch=20861 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=11 delta_bytes=1661 delta_deletes=0 stable_file=dmf_5061750 stable_rows=13678 stable_bytes=2065378 dmf_rows=13678 dmf_bytes=2065378 dmf_packs=2>"] [source="keyspace=4294967295 table_id=3409"] [thread_id=132]
[2025/07/31 06:43:16.194 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:514] ["MergeDelta - Finish, delta is merged, 
old_segment=<segment_id=1 epoch=20861 range=[-9223372036854775808,9223372036854775807) abandoned=true next_segment_id=0 delta_rows=11 delta_bytes=1661 delta_deletes=0 stable_file=dmf_5061750 stable_rows=13678 stable_bytes=2065378 dmf_rows=13678 dmf_bytes=2065378 dmf_packs=2> 
new_segment=<segment_id=1 epoch=20862 range=[-9223372036854775808,9223372036854775807)                next_segment_id=0 delta_rows=0 delta_bytes=0 delta_deletes=0 stable_file=dmf_5062093 stable_rows=13679 stable_bytes=2065529 dmf_rows=13679 dmf_bytes=2065529 dmf_packs=2>"] [source="keyspace=4294967295 table_id=3409"] [thread_id=132]

[2025/07/31 06:43:16.195 +08:00] [INFO] [DeltaMergeStore_InternalBg.cpp:821] ["GC - Trigger MergeDelta, compact_reason=TooManyInvalidVersion segment=<segment_id=1 epoch=22809 range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=4294967295 table_id=3704"] [thread_id=132]
[2025/07/31 06:43:16.195 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:381] ["MergeDelta - Begin, reason=BackgroundGCThread safe_point=459755685378523136 segment=<segment_id=1 epoch=22809 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=9 delta_bytes=1350 delta_deletes=0 stable_file=dmf_5061751 stable_rows=7050 stable_bytes=1057500 dmf_rows=7050 dmf_bytes=1057500 dmf_packs=1>"] [source="keyspace=4294967295 table_id=3704"] [thread_id=132]
[2025/07/31 06:43:16.219 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:514] ["MergeDelta - Finish, delta is merged, 
old_segment=<segment_id=1 epoch=22809 range=[-9223372036854775808,9223372036854775807) abandoned=true next_segment_id=0 delta_rows=9 delta_bytes=1350 delta_deletes=0 stable_file=dmf_5061751 stable_rows=7050 stable_bytes=1057500 dmf_rows=7050 dmf_bytes=1057500 dmf_packs=1> 
new_segment=<segment_id=1 epoch=22810 range=[-9223372036854775808,9223372036854775807)                next_segment_id=0 delta_rows=0 delta_bytes=0 delta_deletes=0 stable_file=dmf_5062094 stable_rows=7052 stable_bytes=1057800 dmf_rows=7052 dmf_bytes=1057800 dmf_packs=1>"] [source="keyspace=4294967295 table_id=3704"] [thread_id=132]

4. What is your TiFlash version? (Required)

v7.5.6

Start by tracing the background GC at DeltaMergeStore_InternalBg.cpp:821 and the merge flow at DeltaMergeStore_InternalSegment.cpp:381 and :514, using the supplied logs as the initial reproduction evidence. Done should mean the excessive read/write amplification is reproduced and its background GC trigger or merge behavior is corrected and verified against comparable logs.