pingcap / pingcap/tidb

ingestor: failed Pebble opens leak disk health checking goroutines

Open
#71,107 1 comment 0 reactions 0 assignees View on GitHub
affects-25.10 affects-26.3 affects-8.1 affects-8.5 component/lightning severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

TiDB's local ingest backend opens Pebble databases with a nil `pebble.Options.FS`, so Pebble installs its default disk-health-checking filesystem. If `pebble.Open` fails after a filesystem metadata operation has started the health-check ticker, the returned DB is nil and TiDB has no handle through which to close the filesystem.

Using TiDB's pinned Pebble dependency (`v1.1.4-0.20250120151818-5dd133a1e6fb`):

1. Repeatedly call `pebble.Open` 100 times for each of these expected failure paths:
- the parent path is a regular file;
- another DB holds the directory lock;
- the database exists and `ErrorIfExists` is enabled.
2. Count goroutines whose top function is `github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1`.
3. Wait longer than the ticker interval and run GC.

Each failed open leaves one disk-health-checking goroutine behind. Across the three cases, 300 failed opens leave 300 goroutines. They remain after waiting and GC.

TiDB has two production `pebble.Open` call sites in `pkg/ingestor/ingestctrl/engine_mgr.go`, used by local ingest engines and the duplicate-detection DB. Repeated import or DDL ingest failures can therefore accumulate these goroutines in a long-running TiDB process.

This is the same ownership problem fixed upstream by cockroachdb/pebble#5787, but TiDB's pinned Pebble revision does not contain that change.

### 2. What did you expect to see? (Required)

When a local ingest Pebble DB fails to open, all filesystem resources created for that attempt should be released. Repeated failures should not increase the number of disk-health-checking goroutines. Successful DB opens must retain disk-health monitoring until `DB.Close`.

### 3. What did you see instead (Required)

Every failed open can leave one `diskHealthCheckingFS.startTickerLocked.func1` goroutine. The goroutine keeps the filesystem wrapper reachable, so GC cannot reclaim it. At very large counts, goroutine profile collection can also cause long non-GC process pauses; pingcap/tidb#62080 reduces repeated profile collection but does not fix this leak.

### 4. What is your TiDB version? (Required)

Reproduced with the Pebble revision pinned by TiDB master and release-8.5:

```text
github.com/cockroachdb/pebble v1.1.4-0.20250120151818-5dd133a1e6fb
```

The issue was also observed on TiDB v8.5.6.

Related upstream change: https://github.com/cockroachdb/pebble/pull/5787

Contributor guide

Open the contributing guide

Research direction

Start in pkg/ingestor/ingestctrl/engine_mgr.go at the two production pebble.Open call sites, then compare the ownership fix in cockroachdb/pebble#5787. Reproduce the listed failed-open paths and profile disk-health-checking goroutines; done means failed opens release their filesystem resources while successful opens retain monitoring until DB.Close.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.