tikv / tikv/pd

TestProxyClientIsCancelledQuicklyOnServerShutdown returns EOF

Open
#10,547 10 comments 0 reactions 0 assignees View on GitHub
type/ci
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Flaky Test

### Which jobs are failing
```text
--- FAIL: TestTSOProxyTestSuite/TestProxyClientIsCancelledQuicklyOnServerShutdown (20.38s)
tso_proxy_test.go:142:
Error Trace: /home/runner/work/pd/pd/tests/server/tso/tso_proxy_test.go:142
/home/runner/work/pd/pd/tests/server/tso/tso_proxy_test.go:170
Error: Received unexpected error:
EOF
Test: TestTSOProxyTestSuite/TestProxyClientIsCancelledQuicklyOnServerShutdown
[2026/04/01 06:39:12.821 +00:00] [INFO] [server.go:260] ["PD config"] [enable-keyspace-group=false] [config="{\"client-urls\":\"http://127.0.0.1:40855\",\"peer-urls\":\"http://127.0.0.1:37041\",\"advertise-client-urls\":\"http://127.0.0.1:40855\",\"advertise-peer-urls\":\"http://127.0.0.1:37041\",\"name\":\"pd1\",\"data-dir\":\"/tmp/pd_tests48981833\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"pd1=http://127.0.0.1:37041,pd2=http://127.0.0.1:34263\",\"initial-cluster-state\":\"new\",\"initial-cluster-token\":\"pd-cluster\",\"join\":\"\",\"lease\":5,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null,\"error-output-path\":\"\"},\"max-concurrent-tso-proxy-streamings\":5000,\"tso-proxy-recv-from-client-timeout\":\"1h0m0s\",\"tso-save-interval\":\"5s\",\"tso-update-physical-interval\":\"10s\",\"enable-local-tso\":false,\"metric\":{\"job\":\"pd1\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":64,\"max-pending-peer-count\":64,\"max-merge-region-size\":54,\"max-merge-region-keys\":0,\"split-merge-interval\":\"1h0m0s\",\"switch-witness-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"enable-cross-table-merge\":\"true\",\"patrol-region-interval\":\"10ms\",\"max-store-down-time\":\"30m0s\",\"max-store-preparing-time\":\"48h0m0s\",\"leader-schedule-limit\":4,\"leader-schedule-policy\":\"count\",\"region-schedule-limit\":2048,\"witness-schedule-limit\":4,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"affinity-schedule-limit\":0,\"hot-region-cache-hits-threshold\":3,\"store-limit\":{},\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.7,\"region-score-formula-version\":\"v2\",\"scheduler-max-waiting-operator\":5,\"enable-remove-down-replica\":\"true\",\"enable-replace-offline-replica\":\"true\",\"enable-make-up-replica\":\"true\",\"enable-remove-extra-replica\":\"true\",\"enable-location-replacement\":\"true\",\"enable-debug-metrics\":\"false\",\"enable-joint-consensus\":\"true\",\"enable-tikv-split-region\":\"true\",\"enable-heartbeat-breakdown-metrics\":\"true\",\"enable-heartbeat-concurrent-runner\":\"false\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"hot-region\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"evict-slow-store\",\"args\":null,\"disable\":false,\"args-payload\":\"\"},{\"type\":\"evict-stopping-store\",\"args\":null,\"disable\":false,\"args-payload\":\"\"}],\"hot-regions-write-interval\":\"10m0s\",\"hot-regions-reserved-days\":7,\"max-movable-hot-peer-size\":512,\"enable-diagnostic\":\"true\",\"enable-witness\":\"false\",\"slow-store-evicting-affected-store-ratio-threshold\":0.3,\"store-limit-version\":\"v1\",\"patrol-region-worker-count\":1,\"max-affinity-merge-region-size\":256},\"replication\":{\"max-replicas\":3,\"location-labels\":\"\",\"strictly-match-label\":\"false\",\"enable-placement-rules\":\"true\",\"enable-placement-rules-cache\":\"false\",\"isolation-level\":\"\"},\"pd-server\":{\"use-region-storage\":\"true\",\"max-gap-reset-ts\":\"24h0m0s\",\"key-type\":\"table\",\"runtime-services\":\"\",\"metric-storage\":\"\",\"dashboard-address\":\"auto\",\"flow-round-by-digit\":3,\"min-resolved-ts-persistence-interval\":\"1s\",\"server-memory-limit\":0,\"server-memory-limit-gc-trigger\":0.7,\"enable-gogc-tuner\":\"false\",\"gc-tuner-threshold\":0.6,\"block-safe-point-v1\":\"false\"},\"cluster-version\":\"0.0.0\",\"labels\":{},\"quota-backend-bytes\":\"8GiB\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"tick-interval\":\"500ms\",\"election-interval\":\"3s\",\"enable-prevote\":true,\"max-request-bytes\":157286400,\"security\":{\"cacert-path\":\"\",\"cert-path\":\"\",\"key-path\":\"\",\"cert-allowed-cn\":null,\"SSLCABytes\":null,\"SSLCertBytes\":null,\"SSLKEYBytes\":null,\"redact-info-log\":false,\"encryption\":{\"data-encryption-method\":\"plaintext\",\"data-key-rotation-period\":\"168h0m0s\",\"master-key\":{\"type\":\"plaintext\",\"key-id\":\"\",\"region\":\"\",\"endpoint\":\"\",\"path\":\"\"}}},\"label-property\":null,\"dashboard\":{\"tidb-cacert-path\":\"\",\"tidb-cert-path\":\"\",\"tidb-key-path\":\"\",\"public-path-prefix\":\"\",\"temp-dir\":\"\",\"internal-proxy\":false,\"enable-telemetry\":false,\"enable-experimental\":false,\"disable-custom-prom-addr\":false},\"replication-mode\":{\"replication-mode\":\"majority\",\"dr-auto-sync\":{\"label-key\":\"\",\"primary\":\"\",\"dr\":\"\",\"primary-replicas\":0,\"dr-replicas\":0,\"wait-store-timeout\":\"1m0s\",\"wait-recover-timeout\":\"0s\",\"pause-region-split\":\"false\"}},\"keyspace\":{\"pre-alloc\":null,\"wait-region-split\":true,\"wait-region-split-timeout\":\"30s\",\"check-region-split-interval\":\"50ms\"},\"micro-service\":{\"enable-scheduling-fallback\":\"true\",\"enable-tso-dynamic-switching\":\"false\",\"enable-resource-manager-fallback\":\"true\"},\"controller\":{\"degraded-mode-wait-duration\":\"0s\",\"ltb-max-wait-duration\":\"30s\",\"ltb-token-rpc-max-delay\":\"1s\",\"request-unit\":{\"read-base-cost\":0.125,\"read-per-batch-base-cost\":0.5,\"read-cost-per-byte\":0.0000152587890625,\"write-base-cost\":1,\"write-per-batch-base-cost\":1,\"write-cost-per-byte\":0.0009765625,\"read-cpu-ms-cost\":0.3333333333333333},\"enable-controller-trace-log\":\"false\"},\"metering\":{}}"]
[2026/04/01 06:39:12.834 +00:00] [INFO] [apiutil.go:462] ["register REST path"] [path=/pd/api/v1]
[2026/04/01 06:39:12.835 +00:00] [INFO] [apiutil.go:462] ["register REST path"] [path=/pd/api/v2/]
[2026/04/01 06:39:12.835 +00:00] [INFO] [apiutil.go:462] ["register REST path"] [path=/dashboard/]
[2026/04/01 06:39:12.835 +00:00] [INFO] [registry.go:94] ["restful PD registered successfully"] [prefix=pd1] [service-name=MetaStorage]
[2026/04/01 06:39:12.840 +00:00] [WARN] [util.go:60] ["go runtime collectors have already registered"] [error="duplicate metrics collector registration attempted"]
```

### New CI link
https://github.com/tikv/pd/actions/runs/23835385547/job/69477952389
https://github.com/tikv/pd/actions/runs/23844872837/job/69509557866

### Reason for failure (if possible)

### Anything else
group_key: tests/server/tso|TestTSOProxyTestSuite/TestProxyClientIsCancelledQuicklyOnServerShutdown
pr_numbers: 10178, 10453

test:TestProxyClientIsCancelledQuicklyOnServerShutdown
package:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.