cockroachdb / cockroachdb/cockroach
Detect Leaked Cloud Storage Handles
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
While investigating #134618 I discovered that there are many call paths leaking cloud.ExternalStorage handles. My plan is to adjust the default test server so that it tracks these allocations and fails the test if any are leaked.
Here's the output from an early draft of my utility when running `TestBackupRestoreSingleNodeLocal`:
```
leaked 11 of 49 cloud.ExternalStorage instances from 7 unique allocation stacks:
found 2 copies of stack:
0: backupinfo.GetBackupManifestIterFactories manifest_handling.go:1680
1: backupccl.loadBackupSQLDescs restore_job.go:614
2: backupccl.(*restoreResumer).doResume restore_job.go:1782
3: backupccl.(*restoreResumer).Resume restore_job.go:1745
4: jobs.(*Registry).stepThroughStateMachine.func2 registry.go:1639
5: jobs.(*Registry).stepThroughStateMachine registry.go:1640
6: jobs.(*Registry).runJob adopt.go:446
7: jobs.(*StartableJob).Start.func2 jobs.go:840
8: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
9: runtime.goexit asm_amd64.s:1695
found 1 copies of stack:
0: backupinfo.GetBackupManifestIterFactories manifest_handling.go:1680
1: backupccl.getBackupDetailAndManifest backup_job.go:1871
2: backupccl.(*backupResumer).Resume.func2 backup_job.go:691
3: sql.(*InternalDB).Txn.func1 internal.go:1937
4: sql.(*InternalDB).txn.func4 internal.go:2024
5: kv.(*Txn).exec txn.go:1052
6: kv.runTxn db.go:1098
7: kv.(*DB).TxnWithAdmissionControl db.go:1061
8: kv.(*DB).Txn db.go:1036
9: sql.(*InternalDB).txn internal.go:2011
10: sql.(*InternalDB).Txn internal.go:1938
11: backupccl.(*backupResumer).Resume backup_job.go:690
12: jobs.(*Registry).stepThroughStateMachine.func2 registry.go:1639
13: jobs.(*Registry).stepThroughStateMachine registry.go:1640
14: jobs.(*Registry).runJob adopt.go:446
15: jobs.(*StartableJob).Start.func2 jobs.go:840
16: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
17: runtime.goexit asm_amd64.s:1695
found 1 copies of stack:
0: backupdest.MakeBackupDestinationStores incrementals.go:135
1: backupccl.doRestorePlan restore_planning.go:1743
2: backupccl.restorePlanHook.func3 restore_planning.go:1430
3: sql.(*hookFnNode).startExec.func1 planhook.go:194
4: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
5: runtime.goexit asm_amd64.s:1695
found 1 copies of stack:
0: backupccl.(*restoreResumer).doResume restore_job.go:1803
1: backupccl.(*restoreResumer).Resume restore_job.go:1745
2: jobs.(*Registry).stepThroughStateMachine.func2 registry.go:1639
3: jobs.(*Registry).stepThroughStateMachine registry.go:1640
4: jobs.(*Registry).runJob adopt.go:446
5: jobs.(*StartableJob).Start.func2 jobs.go:840
6: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
7: runtime.goexit asm_amd64.s:1695
found 2 copies of stack:
0: backupinfo.GetBackupManifestIterFactories manifest_handling.go:1680
1: backupccl.restore restore_job.go:362
2: backupccl.restoreWithRetry restore_job.go:188
3: backupccl.(*restoreResumer).doResume restore_job.go:1955
4: backupccl.(*restoreResumer).Resume restore_job.go:1745
5: jobs.(*Registry).stepThroughStateMachine.func2 registry.go:1639
6: jobs.(*Registry).stepThroughStateMachine registry.go:1640
7: jobs.(*Registry).runJob adopt.go:446
8: jobs.(*StartableJob).Start.func2 jobs.go:840
9: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
10: runtime.goexit asm_amd64.s:1695
found 2 copies of stack:
0: backupinfo.GetBackupManifestIterFactories manifest_handling.go:1680
1: backupccl.makeBackupMetadata generative_split_and_scatter_processor.go:426
2: backupccl.runGenerativeSplitAndScatter.func1 generative_split_and_scatter_processor.go:461
3: backupccl.runGenerativeSplitAndScatter.Group.GoCtx.func5 ctxgroup.go:163
4: errgroup.(*Group).Go.func1 errgroup.go:78
5: runtime.goexit asm_amd64.s:1695
found 2 copies of stack:
0: backupinfo.GetBackupManifestIterFactories manifest_handling.go:1680
1: backupccl.doRestorePlan restore_planning.go:1880
2: backupccl.restorePlanHook.func3 restore_planning.go:1430
3: sql.(*hookFnNode).startExec.func1 planhook.go:194
4: stop.(*Stopper).RunAsyncTaskEx.func2 stopper.go:498
5: runtime.goexit asm_amd64.s:1695
```
Jira issue: CRDB-44406
Contributor guide
Assessment
This issue has not been assessed yet.