pingcap / pingcap/tidb

20x performance regression going from v6.5.2 to v6.5.3 on K8s

Open
#44,715 5 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

I'm using TiDB, installed on K8s using the v1.4.4 operator, without much customization (I basically followed the guides).

When I upgraded to v6.5.3 today I immediately noticed a 20x slowdown in my DB-heavy workloads.
Downgrading to v6.5.2 fixed the issue.

Peculiarities with my setup:
1. I'm running on top of a Tailscale virtual network.
1. I created the K8s cluster using K0s, with Calico for networking.
1. I have one PD, KV, and DB per region and I force reads to be local by using `set global tidb_replica_read = 'closest-replicas';`.

### 1. Minimal reproduce step (Required)

I don't have a minimal case.

### 2. What did you expect to see? (Required)

My particular workload should have a sustained throughput of ~7.5e3 QPS / region / worker, and when I upgraded to 6.5.3 it dropped to ~3e2.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.