ingestor: failed Pebble opens leak disk health checking goroutines
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
TiDB's local ingest backend opens Pebble databases with a nil `pebble.Options.FS`, so Pebble installs its default disk-health-checking filesystem. If `pebble.Open` fails after a filesystem metadata operation has started the health-check ticker, the returned DB is nil and TiDB has no handle through which to close the filesystem.
Using TiDB's pinned Pebble dependency (`v1.1.4-0.20250120151818-5dd133a1e6fb`):
1. Repeatedly call `pebble.Open` 100 times for each of these expected failure paths:
- the parent path is a regular file;
- another DB holds the directory lock;
- the database exists and `ErrorIfExists` is enabled.
2. Count goroutines whose top function is `github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1`.
3. Wait longer than the ticker interval and run GC.
Each failed open leaves one disk-health-checking goroutine behind. Across the three cases, 300 failed opens leave 300 goroutines. They remain after waiting and GC.
TiDB has two production `pebble.Open` call sites in `pkg/ingestor/ingestctrl/engine_mgr.go`, used by local ingest engines and the duplicate-detection DB. Repeated import or DDL ingest failures can therefore accumulate these goroutines in a long-running TiDB process.
This is the same ownership problem fixed upstream by cockroachdb/pebble#5787, but TiDB's pinned Pebble revision does not contain that change.
### 2. What did you expect to see? (Required)
When a local ingest Pebble DB fails to open, all filesystem resources created for that attempt should be released. Repeated failures should not increase the number of disk-health-checking goroutines. Successful DB opens must retain disk-health monitoring until `DB.Close`.
### 3. What did you see instead (Required)
Every failed open can leave one `diskHealthCheckingFS.startTickerLocked.func1` goroutine. The goroutine keeps the filesystem wrapper reachable, so GC cannot reclaim it. At very large counts, goroutine profile collection can also cause long non-GC process pauses; pingcap/tidb#62080 reduces repeated profile collection but does not fix this leak.
### 4. What is your TiDB version? (Required)
Reproduced with the Pebble revision pinned by TiDB master and release-8.5:
```text
github.com/cockroachdb/pebble v1.1.4-0.20250120151818-5dd133a1e6fb
```
The issue was also observed on TiDB v8.5.6.
Related upstream change: https://github.com/cockroachdb/pebble/pull/5787
Contributor guide
Research direction
Start in pkg/ingestor/ingestctrl/engine_mgr.go at the two production pebble.Open call sites, then compare the ownership fix in cockroachdb/pebble#5787. Reproduce the listed failed-open paths and profile disk-health-checking goroutines; done means failed opens release their filesystem resources while successful opens retain monitoring until DB.Close.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100