pingcap / pingcap/tiproxy

Flaky test TestForwardUntilError

Open
#1,222 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

severity/minor type/bug
Dominant language
Go
Stars
73
Forks
41
Avg merge
21h 3m
Merged PRs (30d)
21

Description

panic: test timed out after 10m0s
	running tests:
		TestForwardUntilError (9m52s)
goroutine 6491 [running]:
testing.(*M).startAlarm.func1()
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:2682 +0x359
created by time.goFunc
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/time/sleep.go:215 +0x2d
goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0000b6700, {0x9f7d9f?, 0x0?}, 0xa2b900)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:2005 +0x485
testing.runTests.func1(0xc0000b6700)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:2477 +0x3e
testing.tRunner(0xc0000b6700, 0xc00003fc70)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:1934 +0xea
testing.runTests(0xc000012c48, {0xe823a0, 0x24, 0x24}, {0x7?, 0xc00020dc80?, 0xe8b080?})
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:2475 +0x4b4
testing.(*M).Run(0xc000288280)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:2337 +0x63a
main.main()
	_testmain.go:137 +0x9b
goroutine 6456 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0x8516d0?, 0x50?)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0xc00033c050)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/sync/waitgroup.go:206 +0x85
github.com/pingcap/tiproxy/pkg/proxy/net.TestForwardUntilError(0xc00011ae00)
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/proxy/net/packetio_test.go:650 +0x1cf
testing.tRunner(0xc00011ae00, 0xa2b900)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/testing/testing.go:1997 +0x465
goroutine 6458 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0x8516d0?, 0xa0?)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/runtime/sema.go:114 +0x2e
sync.(*WaitGroup).Wait(0xc00033c0a0)
	/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.12.linux-amd64/src/sync/waitgroup.go:206 +0x85
github.com/pingcap/tiproxy/pkg/testkit.TestTCPConnWithListener(0xc00011ae00, 0x424b01?, 0xc00039a078, 0xc00039a090, 0x1)
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/testkit/main.go:72 +0x112
github.com/pingcap/tiproxy/pkg/testkit.TestTCPConn(...)
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/testkit/main.go:36
github.com/pingcap/tiproxy/pkg/proxy/net.testTCPConn(0xc00011ae00, 0xa2bb70, 0xc000388240, 0x1)
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/proxy/net/packetio_test.go:35 +0x112
github.com/pingcap/tiproxy/pkg/proxy/net.TestForwardUntilError.func2()
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/proxy/net/packetio_test.go:636 +0xbc
github.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithLogger.func1()
	/home/prow/go/src/github.com/pingcap/tiproxy/lib/util/waitgroup/waitgroup.go:41 +0x53
created by github.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithLogger in goroutine 6456
	/home/prow/go/src/github.com/pingcap/tiproxy/lib/util/waitgroup/waitgroup.go:27 +0x90
goroutine 6464 [chan receive, 9 minutes]:
github.com/pingcap/tiproxy/pkg/proxy/net.TestForwardUntilError.func2.2(0xae0cb0?, 0xc0000ce370)
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/proxy/net/packetio_test.go:645 +0xf5
github.com/pingcap/tiproxy/pkg/proxy/net.testTCPConn.func2(0xc00011ae00, {0xae0cb0?, 0xc000124010?})
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/proxy/net/packetio_test.go:43 +0x63
github.com/pingcap/tiproxy/pkg/testkit.TestTCPConnWithListener.func3()
	/home/prow/go/src/github.com/pingcap/tiproxy/pkg/testkit/main.go:67 +0x116
github.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithLogger.func1()
	/home/prow/go/src/github.com/pingcap/tiproxy/lib/util/waitgroup/waitgroup.go:41 +0x53
created by github.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithLogger in goroutine 6458
	/home/prow/go/src/github.com/pingcap/tiproxy/lib/util/waitgroup/waitgroup.go:27 +0x90

See https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/pingcap_tiproxy/1221/pull-unit-test/2099717962410758144

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with TestForwardUntilError in pkg/proxy/net/packetio_test.go, especially the lines shown in the stack trace, and inspect the related helpers in pkg/testkit/main.go and lib/util/waitgroup/waitgroup.go. Reproduce it with the targeted Go test and repeated runs, then confirm the test completes reliably without the ten-minute timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.