test options do not work for compiled binaries
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 280
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 2
Description
### Godog.BindCommandLineFlags in init does not work for binaries
When using test flags like `-coverprofile` they are not honored when trying a godog application which has been created by `go test -c ... -o ...`
#### happy case
Using the a test as suggested by the README:
```go
package foo
import (
"os"
"testing"
"github.com/cucumber/godog"
"github.com/cucumber/godog/colors"
"github.com/spf13/pflag" // godog v0.11.0 and later
)
var opts = godog.Options{
Output: colors.Colored(os.Stdout),
Format: "progress", // can define default values
}
func init() {
godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
}
func TestMain(m *testing.M) {
pflag.Parse()
opts.Paths = pflag.Args()
status := godog.TestSuite{
Name: "godogs",
Options: &opts,
}.Run()
// Optional: Run `testing` package's logic besides godog.
if st := m.Run(); st > status {
status = st
}
os.Exit(status)
}
```
Running this directly is working:
```
$ go test -coverprofile foo.txt godog_test.go
ok command-line-arguments 0.060s coverage: [no statements] [no tests to run]
```
Running the compiled variant:
```
$ go test -c -coverprofile foo.txt godog_test.go
$ ./foo.test -coverprofile foo.txt godog_test.go
./foo.test -coverprofile foo.txt godog_test.go
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
-c, --godog.concurrency int run the test suite with concurrency (default 1)
-d, --godog.definitions print all available step definitions
-f, --godog.format string will write a report according to the selected formatter
usage:
-f
```
Notice that the help file does not show any `--test.coverprofile` as suggestion.
### ✨ Do you have a proposal for making it better?
It seems *not* having `godog.BindCommandLineFlags("godog.", &opts)` in a `init()` function and placing it just before `pflag.Parse()` is enough. And seems to work for test binaries as well as with directly ran `go test` applications
Continuing with the code given above
```diff
17,20d16
< func init() {
< godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
< }
<
21a18
> godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
```
Rerunning:
```
./foo.test -coverprofile foo.txt godog_test.go
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
-c, --godog.concurrency int run the test suite with concurrency (default 1)
......
--test.bench regexp run only benchmarks matching regexp
--test.benchmem print memory allocations for benchmarks
--test.benchtime d run each benchmark for duration d (default 1s)
--test.blockprofile file write a goroutine blocking profile to file
--test.blockprofilerate rate set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
--test.count n run tests and benchmarks n times (default 1)
--test.coverprofile file write a coverage profile to file
--test.cpu list comma-separated list of cpu counts to run each test with
--test.cpuprofile file write a cpu profile to file
--test.failfast do not start new tests after the first test failure
--test.fuzz regexp run the fuzz test matching regexp
....
```
Still not working, though notice the added `--test-....` options now.
Simply using `--test.coverprofile` now works:
```
rm foo.txt; ./foo.test --test.coverprofile foo.txt > /dev/null ; ls -alh foo.txt
testing: warning: no tests to run
-rw-rw-r-- 1 hvdb hvdb 10 Jan 12 21:43 foo.txt
```
### 📚 Any additional context?
I'm unsure if there was a good reason to have the above line in the `init()` in the first place (?)
----
*This text was originally generated from a [template](https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates), then edited by hand. [You can modify the template here.](https://github.com/cucumber/.github/edit/main/.github/ISSUE_TEMPLATE/developer_experience.md)*
Contributor guide
Research direction
Start with godog.BindCommandLineFlags and the README's TestMain example, comparing flag registration in init with registration immediately before pflag.Parse(). Reproduce both direct go test and go test -c runs, then verify that the compiled binary accepts --test.coverprofile without misparsing it as a godog flag.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100