pingcap / pingcap/ticdc

cancel the workload script cause the resolved-ts spike

Open
#3,971 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

severity/moderate type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

  • run the following scripts with the command ./xxx 20000 0
  • ctrl + c to stop the script
#!/bin/bash

batch_size=$1
sleep_duration=$2

TABLE_NAME="test.bank0"

i=0
while true
do
    sleep "$sleep_duration"

    i=$((i + 1))
    start_ts=$(date +%s%3N)

    mysql -h$HOST -P$PORT -e "DELETE FROM $TABLE_NAME LIMIT $batch_size;"

    end_ts=$(date +%s%3N)
    cost_ms=$((end_ts - start_ts))

    echo "Round $i: delete from $TABLE_NAME finished, cost=${cost_ms} ms"
done

The resolved-ts spikes, and TiKV-CDC endpoint CPU become full

Image Image

This may caused by the stale lock, the resolved-ts drop to 0 after GC

Image

At the same time, the TiKV-CDC endpoint CPU usage is full, this is not expected.

Image Image

tikv:4855bdccc64e7a8551d30ebbbd5be75a42929265

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the DELETE loop against TiDB with the provided script and the referenced TiKV commit, then observe resolved-ts and TiKV-CDC endpoint CPU behavior after Ctrl+C and garbage collection. Trace the stale-lock or resolved-ts transition identified in the report; done means the cause is explained and the endpoint no longer exhibits the unexpected spike or full CPU usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, go, mysql
Domain
databases, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.