microsoft / microsoft/go

Review ultra-verbose `go tool dist test` logs for unexpected skips.

Open
#65 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
431
Forks
44
Avg merge
21h 18m
Merged PRs (30d)
30

Description

Broken off from original issue tracking full coverage vs. upstream: https://github.com/microsoft/go/issues/1

Something we can do to be a little more sure our tests are running properly is to look at the verbose logs, with a modification to make them more verbose and show SKIPs:

Here's what I did to make tests show skips:

diff --git a/src/cmd/dist/test.go b/src/cmd/dist/test.go
index 0c8e2c56bc..1248d9145b 100644
--- a/src/cmd/dist/test.go
+++ b/src/cmd/dist/test.go
@@ -271,7 +271,7 @@ func short() string {
 // defaults as later arguments in the command line.
 func (t *tester) goTest() []string {
 	return []string{
-		"go", "test", "-short=" + short(), "-count=1", t.tags(), t.runFlag(""),
+		"go", "test", "-v", "-short=" + short(), "-count=1", t.tags(), t.runFlag(""),
 	}
 }
 
@@ -350,6 +350,7 @@ func (t *tester) registerStdTest(pkg string, useG3 bool) {
 			}
 			args := []string{
 				"test",
+				"-v",
 				"-short=" + short(),
 				t.tags(),
 				t.timeout(timeoutSec),
@@ -388,6 +389,7 @@ func (t *tester) registerRaceBenchTest(pkg string) {
 			ranGoBench = true
 			args := []string{
 				"test",
+				"-v",
 				"-short=" + short(),
 				"-race",
 				t.timeout(1200), // longer timeout for race with benchmarks

~500 results for 'SKIP': https://gist.github.com/dagood/2bc08e37da295c2022b9196572d378a4
Results with 1 line of context (skip reason): https://gist.github.com/dagood/8105bad6aff4803794a4ed42c265a4fc

500 lines isn't small, but it isn't impossible. There are duplicate skip reasons, which helps.

Unfortunately, there is no baseline from upstream to check against. We need to reason out whether each one is a problem.

This is based on a golang-devs thread: https://groups.google.com/g/golang-dev/c/PNzwZXOe7bQ/m/M43Gl9mVDAAJ.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/cmd/dist/test.go, especially tester.goTest, registerStdTest, and registerRaceBenchTest, and inspect the verbose go tool dist test output described in the issue. Review the linked SKIP logs and their one-line context, grouping duplicate reasons and checking each for an unexpected omission. Done means documenting which skips are problematic; the issue notes there is no upstream baseline, so comparison requires reasoning.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.