cockroachdb / cockroachdb/cockroach

roachtest: backupTPCC leaks a goroutine running tpcc workload

Open
#104,314 1 comment 0 reactions 0 assignees View on GitHub
A-testing C-bug T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

I happened to debug an unrelated issue which required `SIGQUIT`. The following goroutine was running much later after the corresponding `backupTPCC` finished. (Note, the test has a timeout of 1h whereas the goroutine in question exceeds it by a long shot.)

```
1: chan send [1810 minutes] [Created by tests.registerBackup.func4 @ backup.go:726]
runtime proc.go:363 gopark(func(#9359), Pointer(0x50), waitReason(#74))
runtime chan.go:259 chansend(*hchan(#4245), Pointer(#3437), true)
runtime chan.go:145 chansend1(*hchan(#263), Pointer(#96))
tests backup.go:732 registerBackup.func4.2()
runtime asm_amd64.s:1594 goexit()
```

Upon a closer examination, the tpcc workload has no explicit time out [1]. Thus, it's possible for it to continue running long after the test is done; e.g., cluster is reused and subsequent test(s) load tpcc. This can be remedied either by passing `--duration` and/or running it inside context-aware`Monitor`, e.g., [2].

[1] https://github.com/cockroachdb/cockroach/blob/c8c8869d8223e65eb4e6e6c32a9f8680a9e68f36/pkg/cmd/roachtest/tests/backup.go#L728-L733
[2] https://github.com/cockroachdb/cockroach/blob/c8c8869d8223e65eb4e6e6c32a9f8680a9e68f36/pkg/cmd/roachtest/tests/tpcc.go#L249

Jira issue: CRDB-28456

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.