cockroachdb / cockroachdb/cockroach
roachtest: backupTPCC leaks a goroutine running tpcc workload
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
I happened to debug an unrelated issue which required `SIGQUIT`. The following goroutine was running much later after the corresponding `backupTPCC` finished. (Note, the test has a timeout of 1h whereas the goroutine in question exceeds it by a long shot.)
```
1: chan send [1810 minutes] [Created by tests.registerBackup.func4 @ backup.go:726]
runtime proc.go:363 gopark(func(#9359), Pointer(0x50), waitReason(#74))
runtime chan.go:259 chansend(*hchan(#4245), Pointer(#3437), true)
runtime chan.go:145 chansend1(*hchan(#263), Pointer(#96))
tests backup.go:732 registerBackup.func4.2()
runtime asm_amd64.s:1594 goexit()
```
Upon a closer examination, the tpcc workload has no explicit time out [1]. Thus, it's possible for it to continue running long after the test is done; e.g., cluster is reused and subsequent test(s) load tpcc. This can be remedied either by passing `--duration` and/or running it inside context-aware`Monitor`, e.g., [2].
[1] https://github.com/cockroachdb/cockroach/blob/c8c8869d8223e65eb4e6e6c32a9f8680a9e68f36/pkg/cmd/roachtest/tests/backup.go#L728-L733
[2] https://github.com/cockroachdb/cockroach/blob/c8c8869d8223e65eb4e6e6c32a9f8680a9e68f36/pkg/cmd/roachtest/tests/tpcc.go#L249
Jira issue: CRDB-28456
Contributor guide
Assessment
This issue has not been assessed yet.