cucumber / cucumber/godog

test options do not work for compiled binaries

Open
#533 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
280
Avg merge
2d 20h
Merged PRs (30d)
2

Description

### Godog.BindCommandLineFlags in init does not work for binaries

When using test flags like `-coverprofile` they are not honored when trying a godog application which has been created by `go test -c ... -o ...`

#### happy case

Using the a test as suggested by the README:

```go
package foo

import (
"os"
"testing"

"github.com/cucumber/godog"
"github.com/cucumber/godog/colors"
"github.com/spf13/pflag" // godog v0.11.0 and later
)

var opts = godog.Options{
Output: colors.Colored(os.Stdout),
Format: "progress", // can define default values
}

func init() {
godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
}

func TestMain(m *testing.M) {
pflag.Parse()
opts.Paths = pflag.Args()

status := godog.TestSuite{
Name: "godogs",
Options: &opts,
}.Run()

// Optional: Run `testing` package's logic besides godog.
if st := m.Run(); st > status {
status = st
}

os.Exit(status)
}

```

Running this directly is working:

```
$ go test -coverprofile foo.txt godog_test.go
ok command-line-arguments 0.060s coverage: [no statements] [no tests to run]

```

Running the compiled variant:

```
$ go test -c -coverprofile foo.txt godog_test.go
$ ./foo.test -coverprofile foo.txt godog_test.go
./foo.test -coverprofile foo.txt godog_test.go
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
-c, --godog.concurrency int run the test suite with concurrency (default 1)
-d, --godog.definitions print all available step definitions
-f, --godog.format string will write a report according to the selected formatter

usage:
-f

```

Notice that the help file does not show any `--test.coverprofile` as suggestion.

### ✨ Do you have a proposal for making it better?
It seems *not* having `godog.BindCommandLineFlags("godog.", &opts)` in a `init()` function and placing it just before `pflag.Parse()` is enough. And seems to work for test binaries as well as with directly ran `go test` applications

Continuing with the code given above

```diff
17,20d16
< func init() {
< godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
< }
<
21a18
> godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later

```
Rerunning:

```
./foo.test -coverprofile foo.txt godog_test.go
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
-c, --godog.concurrency int run the test suite with concurrency (default 1)
......
--test.bench regexp run only benchmarks matching regexp
--test.benchmem print memory allocations for benchmarks
--test.benchtime d run each benchmark for duration d (default 1s)
--test.blockprofile file write a goroutine blocking profile to file
--test.blockprofilerate rate set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
--test.count n run tests and benchmarks n times (default 1)
--test.coverprofile file write a coverage profile to file
--test.cpu list comma-separated list of cpu counts to run each test with
--test.cpuprofile file write a cpu profile to file
--test.failfast do not start new tests after the first test failure
--test.fuzz regexp run the fuzz test matching regexp
....
```

Still not working, though notice the added `--test-....` options now.

Simply using `--test.coverprofile` now works:

```
rm foo.txt; ./foo.test --test.coverprofile foo.txt > /dev/null ; ls -alh foo.txt
testing: warning: no tests to run
-rw-rw-r-- 1 hvdb hvdb 10 Jan 12 21:43 foo.txt

```

### 📚 Any additional context?

I'm unsure if there was a good reason to have the above line in the `init()` in the first place (?)

----

*This text was originally generated from a [template](https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates), then edited by hand. [You can modify the template here.](https://github.com/cucumber/.github/edit/main/.github/ISSUE_TEMPLATE/developer_experience.md)*

Contributor guide

Open the contributing guide

Research direction

Start with godog.BindCommandLineFlags and the README's TestMain example, comparing flag registration in init with registration immediately before pflag.Parse(). Reproduce both direct go test and go test -c runs, then verify that the compiled binary accepts --test.coverprofile without misparsing it as a godog flag.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.