gotestyourself / gotestyourself/gotestsum

Allow re-running `panic`'ed tests through a runtime configuration flag

Open
#290 10 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
2.7k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

I'm using `gotestsum` to test a terraform provider that frequently has a very flaky acceptance test build (context in https://github.com/terra-farm/terraform-provider-xenorchestra/issues/188). The flakiness is due to tests becoming hung and eventually reaching `go test`'s timeout value. Re-running these tests results in a successful build, but requires a significant amount of manual work to identify what should be rerun.

I've tested a fork of `gotestsum` that allows rerunning `panic`'ed tests (https://github.com/ddelnano/gotestsum/commit/8c70ff05e83bc93e5e042151a4d25343f1dfe4f8) and it appears to work the way I intend it to -- tests that timeout are re-run and result in a passing test build. I created a [dummy test package](https://github.com/terra-farm/terraform-provider-xenorchestra/blob/2501b7d3e4076f88b87e09c511bf8f02fd9f1d89/cmd/testing/parallel/main_test.go) to test this. The test package includes 3 tests and will allow one of the tests to proceed with execution every 10 seconds. If `go test` is given a timeout greater than 30s all three tests pass. If the timeout is between 20s and 30s, then 1 of the tests will timeout and panic.

I used this package to test the behavior of my fork and verified that the result is expected.

```
# Verify that gotestsum re-runs the panic'ed tests and reports a successful build (exit code 0)
ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ ~/code/gotestsum/gotestsum --rerun-fails=2 --packages='github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel' -- -v -count=1 -timeout=21s
✖ cmd/testing/parallel (21.016s)

DONE 3 tests, 1 failure in 21.846s

✓ cmd/testing/parallel (10.031s)

=== Failed
=== FAIL: cmd/testing/parallel TestParallelHangging (unknown)

DONE 2 runs, 4 tests, 1 failure in 32.883s

ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ echo $?
0

# Force test to always fail and verify that gotestsum reports a failure
ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ git diff
diff --git a/cmd/testing/parallel/main_test.go b/cmd/testing/parallel/main_test.go
index b47d341..1c9cfc9 100644
--- a/cmd/testing/parallel/main_test.go
+++ b/cmd/testing/parallel/main_test.go
@@ -38,4 +38,5 @@ func TestParallelHangging(t *testing.T) {
func TestParallelSlightHangging(t *testing.T) {
t.Parallel()
waitForChannel(c)
+ t.Errorf("Failed")
}

ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ ~/code/gotestsum/gotestsum --rerun-fails=2 --packages='github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel' -- -v -count=1 -timeout=21s || echo $?
✖ cmd/testing/parallel (21.008s)

DONE 3 tests, 2 failures in 21.384s

✖ cmd/testing/parallel (10.014s)
✓ cmd/testing/parallel (10.012s)

DONE 2 runs, 5 tests, 3 failures in 42.515s

✖ cmd/testing/parallel (10.011s)

=== Failed
=== FAIL: cmd/testing/parallel TestParallelSlightHangging (20.01s)
main_test.go:41: Failed
panic: test timed out after 21s

goroutine 19 [running]:
testing.(*M).startAlarm.func1()
/usr/lib/go/src/testing/testing.go:2036 +0x8e
created by time.goFunc
/usr/lib/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive]:
testing.tRunner.func1()
/usr/lib/go/src/testing/testing.go:1412 +0x4a5
testing.tRunner(0xc000116680, 0xc000106c78)
/usr/lib/go/src/testing/testing.go:1452 +0x144
testing.runTests(0xc0001300a0?, {0x605400, 0x3, 0x3}, {0x4a6be9?, 0x7f497f542fff?, 0x609ca0?})
/usr/lib/go/src/testing/testing.go:1844 +0x456
testing.(*M).Run(0xc0001300a0)
/usr/lib/go/src/testing/testing.go:1726 +0x5d9
github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain(0x7f49a6674a68?)
/home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:21 +0x5e
main.main()
_testmain.go:53 +0x1d3

goroutine 34 [sleep]:
time.Sleep(0x2540be400)
/usr/lib/go/src/runtime/time.go:195 +0x135
github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain.func1()
/home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:16 +0x2c
created by github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain
/home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:14 +0x54

goroutine 36 [chan receive]:
github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.waitForChannel(...)
/home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:25
github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestParallelHangging(0x0?)
/home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:35 +0x27
testing.tRunner(0xc000116b60, 0x531050)
/usr/lib/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
/usr/lib/go/src/testing/testing.go:1493 +0x35f

=== FAIL: cmd/testing/parallel TestParallelHangging (unknown)

=== FAIL: cmd/testing/parallel TestParallelSlightHangging (re-run 1) (10.01s)
main_test.go:41: Failed

=== FAIL: cmd/testing/parallel TestParallelSlightHangging (re-run 2) (10.01s)
main_test.go:41: Failed

DONE 3 runs, 6 tests, 4 failures in 53.302s
1
```

While working on this, I realized that there is a history on handling this situation (https://github.com/gotestyourself/gotestsum/pull/192 and https://github.com/golang/go/issues/45508), so I know it was intentional that these tests aren't rerun but I'd like to make it possible to enable this behavior behind a new cli flag (`--rerun-fails-rerun-panics` or something similar).

I'm opening this issue to discuss if a PR that implements this would be accepted. If that is the case, I'm happy to take on this work myself.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the behavior demonstrated in cmd/testing/parallel/main_test.go and review gotestsum/pull/192 plus golang/go issue 45508 for the existing decision about panic'ed tests. Trace how the rerun-fails option handles test output and statuses. Done means a new runtime flag can opt into rerunning timed-out tests while preserving the correct success or failure exit status.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.