TiFlash OOM with CH workload and 2 AP threads.

Enhancement

Pod under 48 GB memory limited.

TP thread 60, AP thread 2, exclude Q5 and Q9, TiFlash keep oom. If running with only one AP thread, there is no oom.

go-tpc ch --host lb.perftest-ch-hybrid-001-tps-811005-1-7 -P4000 --warehouses 1000 run -D tpcc -T 60 -t 2 --queries q1,q2,q3,q4,q6,q7,q8,q10,q11,q12,q13,q14,q15,q16,q17,q18,q19,q20,q21,q22 --ignore-error --refresh-conn-wait 10s --time 168h0m0s

[root@tiflash-1-0 ~]# dmesg -T | grep oom | tail -20
[Fri May 27 10:55:27 2022] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Fri May 27 10:55:27 2022] oom_reaper: reaped process 5876 (blackbox_export), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Fri May 27 10:55:27 2022] DynamicThread invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[Fri May 27 10:55:27 2022] [] oom_kill_process+0x2b8/0x524
[Fri May 27 10:55:27 2022] [] mem_cgroup_oom_synchronize+0x3c0/0x424
[Fri May 27 10:55:28 2022] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Fri May 27 10:55:28 2022] DynamicThread invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[Fri May 27 10:55:28 2022] [] oom_kill_process+0x2b8/0x524
[Fri May 27 10:55:28 2022] [] mem_cgroup_oom_synchronize+0x3c0/0x424
[Fri May 27 10:55:28 2022] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Fri May 27 10:55:28 2022] oom_reaper: reaped process 6015 (node_exporter), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Fri May 27 10:55:28 2022] DynamicThread invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[Fri May 27 10:55:28 2022] [] oom_kill_process+0x2b8/0x524
[Fri May 27 10:55:28 2022] [] mem_cgroup_oom_synchronize+0x3c0/0x424
[Fri May 27 10:55:29 2022] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Fri May 27 10:55:29 2022] DynamicThread invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
[Fri May 27 10:55:29 2022] [] oom_kill_process+0x2b8/0x524
[Fri May 27 10:55:29 2022] [] mem_cgroup_oom_synchronize+0x3c0/0x424
[Fri May 27 10:55:29 2022] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[Fri May 27 10:55:30 2022] oom_reaper: reaped process 6164 (TiFlashMain), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Start by reproducing the reported go-tpc CH workload with 60 TP threads and 2 AP threads on the 48 GB-limited pod. Inspect the TiFlash process and the dmesg OOM records, comparing against a single AP thread. Done means the workload no longer causes TiFlash to be killed, with the relevant memory behavior understood and covered by an appropriate verification.