[Bug] Next-gen integration tests race on shared TiDB slow log
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Description
Next-gen integration tests can fail while starting multiple TiDB instances
because all instances use the same default relative slow-query log path,
tidb-slow.log.
A TiDB instance may exit with:
[FATAL] unexpected error
can't write to log file: open tidb-slow.log: no such file or directory
The subsequent health check repeatedly receives connection refused and exits
with code 2, causing Jenkins to abort the remaining matrix branches.
Context
This was observed in both storage integration jobs for PR #5781:
- Heavy next-gen build #527, group G11:
https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_storage_integration_heavy_next_gen/527/ - Light next-gen build #529, group G04:
https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_storage_integration_light_next_gen/529/
The failure happens before TiCDC is started, so it is unrelated to the tested
TiCDC behavior.
Root Cause
start_tidb_cluster_nextgen starts the two upstream keyspace TiDB instances
concurrently. Their main log files are unique, but their slow-query log path
is not configured, so both use tidb-slow.log.
The pingcap/log file initialization performs a non-atomic
Stat/Open or Create/Remove sequence. Concurrent initialization of the same
file can therefore cause one process to remove the file after another process
has successfully checked it, making the latter Open fail with ENOENT.
Proposed Solution
Configure a unique absolute slow-query log path for every TiDB instance started
by the next-gen integration test framework.
Also add a regression test that verifies each TiDB launch command receives a
distinct slow-query log path.
Acceptance Criteria
- Every TiDB instance started by
start_tidb_cluster_nextgenuses a unique
slow-query log file. - Existing main log paths and startup behavior remain unchanged.
- A reusable automated test covers the generated slow-query log paths.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the start_tidb_cluster_nextgen entry point and trace how each TiDB launch command is assembled, especially the slow-query log argument. Review the existing next-gen integration test setup before adding a reusable check that generated paths are distinct and absolute. Done means concurrent instances retain their existing main logs and startup behavior without sharing tidb-slow.log.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100