gotestyourself / gotestyourself/gotestsum

Test cases passed but gotestsum print out Failed in summary

Open
#400 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

We tried to adopt gotestsum to https://github.com/karmada-io/karmada repo, and in both CI and local tests, the summary printed by gotestsum returned FAILED even when all test cases passed. Rerun test cases again using `go test` command and it passed normally.

This is the stdout printed by gotestsum in CI (Github Actions)
ref: https://github.com/karmada-io/karmada/actions/runs/8752542558/job/24020602781?pr=4850#step:4:8673

This is the stdout printed by gotestsum in the local
```console
$ gotestsum -- -v -run=TestGetMinTolerationTime pkg/util/helper/*.go
✓ command-line-arguments (478ms)

=== Failed
=== FAIL: command-line-arguments TestGetMinTolerationTime (unknown)
-1ns=== RUN TestGetMinTolerationTime/no_usedTolerations
0s=== RUN TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations
59.999932917s=== RUN TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil
-1ns=== RUN TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil
-1ns=== RUN TestGetMinTolerationTime/find_the_latest_trigger_time
59.999909167s=== RUN TestGetMinTolerationTime/trigger_time_is_up
0s--- PASS: TestGetMinTolerationTime (0.00s)
--- PASS: TestGetMinTolerationTime/no_noExecuteTaints (0.00s)
--- PASS: TestGetMinTolerationTime/no_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/find_the_latest_trigger_time (0.00s)
--- PASS: TestGetMinTolerationTime/trigger_time_is_up (0.00s)

=== FAIL: command-line-arguments TestGetMinTolerationTime/no_noExecuteTaints (unknown)
-1ns=== RUN TestGetMinTolerationTime/no_usedTolerations
0s=== RUN TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations
59.999932917s=== RUN TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil
-1ns=== RUN TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil
-1ns=== RUN TestGetMinTolerationTime/find_the_latest_trigger_time
59.999909167s=== RUN TestGetMinTolerationTime/trigger_time_is_up
0s--- PASS: TestGetMinTolerationTime (0.00s)
--- PASS: TestGetMinTolerationTime/no_noExecuteTaints (0.00s)
--- PASS: TestGetMinTolerationTime/no_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/find_the_latest_trigger_time (0.00s)
--- PASS: TestGetMinTolerationTime/trigger_time_is_up (0.00s)

DONE 2 tests, 2 failures in 3.979s
```

gotestsum printed FAIL, while rerun using `go test`

```console
$ go test -v -run=TestGetMinTolerationTime pkg/util/helper/*.go
=== RUN TestGetMinTolerationTime
=== RUN TestGetMinTolerationTime/no_noExecuteTaints
-1ns=== RUN TestGetMinTolerationTime/no_usedTolerations
0s=== RUN TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations
59.999899792s=== RUN TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil
-1ns=== RUN TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil
-1ns=== RUN TestGetMinTolerationTime/find_the_latest_trigger_time
59.999854833s=== RUN TestGetMinTolerationTime/trigger_time_is_up
0s--- PASS: TestGetMinTolerationTime (0.00s)
--- PASS: TestGetMinTolerationTime/no_noExecuteTaints (0.00s)
--- PASS: TestGetMinTolerationTime/no_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/with_noExecuteTaints_and_usedTolerations (0.00s)
--- PASS: TestGetMinTolerationTime/usedTolerantion.TolerationSeconds_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/noExecuteTaints.TimeAdded_is_nil (0.00s)
--- PASS: TestGetMinTolerationTime/find_the_latest_trigger_time (0.00s)
--- PASS: TestGetMinTolerationTime/trigger_time_is_up (0.00s)
PASS
ok command-line-arguments 0.477s
```

Is this something wrong with gotestsum or did I miss setting any flags?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported `gotestsum -- -v -run=TestGetMinTolerationTime pkg/util/helper/*.go` command and compare its output with `go test`. Trace how gotestsum parses the test output and builds the summary; done means the passing test and subtests no longer appear as failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.