flaky integration test: TestIntegration/7th-rollout-unblocked-after-pollers-die-version-deleted

    --- FAIL: TestIntegration/7th-rollout-unblocked-after-pollers-die-version-deleted (32.83s)

We have a flaky integration test that periodically fails. A sampling of log records from the test output (de-duplicated):

2026-08-19T22:36:28Z	DEBUG	Running Reconcile loop	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"conditions-temporal-state-fetch-failed","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "conditions-temporal-state-fetch-failed", "reconcileID": "e41b070e-2cfd-49cd-992b-aebfa64596cb"}
2026-08-19T22:36:28Z	DEBUG	events	Unable to get worker deployment state: unable to describe worker deployment test-integration-20260819222923/conditions-temporal-state-fetch-failed: Namespace does-not-exist is not found.	{"type": "Warning", "object": {"kind":"WorkerDeployment","namespace":"test-integration-20260819222923","name":"conditions-temporal-state-fetch-failed","uid":"e28c5c8f-baf1-4fa1-8b5c-dc306c8cac94","apiVersion":"temporal.io/v1alpha1","resourceVersion":"945"}, "reason": "TemporalStateFetchFailed"}
2026-08-19T22:36:28Z	ERROR	unable to create TemporalClient	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"conditions-client-creation-failed","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "conditions-client-creation-failed", "reconcileID": "0f3a27cd-52ad-4d33-bb3c-4e04ecd32a74", "error": "failed reaching server: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:1: connect: connection refused\""}
github.com/temporalio/temporal-worker-controller/internal/controller.(*WorkerDeploymentReconciler).Reconcile
	/home/runner/work/temporal-worker-controller/temporal-worker-controller/internal/controller/worker_controller.go:282
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:221
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:478
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:312

repeated many times followed by:

2026-08-19T22:36:28Z	DEBUG	events	Unable to create Temporal client for localhost:1:temporaltest-814240: failed reaching server: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:1: connect: connection refused"	{"type": "Warning", "object": {"kind":"WorkerDeployment","namespace":"test-integration-20260819222923","name":"conditions-client-creation-failed","uid":"a20b3d1b-9cc0-4df6-b3fb-95d84688a235","apiVersion":"temporal.io/v1alpha1","resourceVersion":"1110"}, "reason": "TemporalClientCreationFailed"}
2026-08-19T22:36:28Z	ERROR	Reconciler error	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"conditions-client-creation-failed","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "conditions-client-creation-failed", "reconcileID": "0f3a27cd-52ad-4d33-bb3c-4e04ecd32a74", "error": "failed reaching server: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:1: connect: connection refused\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:494
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:437
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.24.0/pkg/internal/controller/controller.go:312
2026-08-19T22:36:28Z	DEBUG	Running Reconcile loop	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"6th-rollout-unblocked-after-pollers-die-max-ctrlr-versions","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "6th-rollout-unblocked-after-pollers-die-max-ctrlr-versions", "reconcileID": "64a7d483-23ba-4d5d-931b-1a276d469301"}
<snip>

With the following at the tail of the log:

2026-08-19T22:36:34Z	DEBUG	Running Reconcile loop	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"wrt-scaletargetref-empty-object-sentinel","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "wrt-scaletargetref-empty-object-sentinel", "reconcileID": "d681a2aa-6fb6-439a-988c-e3418eeb08ef"}
2026-08-19T22:36:34Z	INFO	Detected rollback scenario using LastCurrentTime. Warning: Auto-upgrade workflows that upgraded from a previous version to the current version may fail during this rollback, as they may not handle downgrades properly. Monitor workflow executions for failures.	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment", "WorkerDeployment": {"name":"wrt-scaletargetref-empty-object-sentinel","namespace":"test-integration-20260819222923"}, "namespace": "test-integration-20260819222923", "name": "wrt-scaletargetref-empty-object-sentinel", "reconcileID": "d681a2aa-6fb6-439a-988c-e3418eeb08ef", "targetBuildID": "v1.0-8578", "lastCurrentTime": "2026-08-19T22:35:29Z"}
    deletion_integration_test.go:478: Verified: drained v1.0 (buildID=v1.0-9d9b) was pruned from the Temporal server on sunset
2026-08-19T22:36:34Z	INFO	Stopping and waiting for non leader election runnables
2026-08-19T22:36:34Z	INFO	Stopping and waiting for leader election runnables
2026-08-19T22:36:34Z	INFO	Stopping and waiting for warmup runnables
2026-08-19T22:36:34Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment"}
2026-08-19T22:36:34Z	INFO	All workers finished	{"controller": "workerdeployment", "controllerGroup": "temporal.io", "controllerKind": "WorkerDeployment"}
2026-08-19T22:36:34Z	INFO	Stopping and waiting for caches
2026-08-19T22:36:34Z	INFO	Stopping and waiting for webhooks
2026-08-19T22:36:34Z	INFO	Stopping and waiting for HTTP servers
2026-08-19T22:36:34Z	INFO	controller-runtime.metrics	Shutting down metrics server with timeout of 1 minute
2026-08-19T22:36:34Z	INFO	Wait completed, proceeding to shutdown the manager
<snip>
...
{"time":"2026-08-19T22:36:39.10144397Z","level":"WARN","msg":"Failed to poll for task.","Namespace":"temporaltest-814240","TaskQueue":"manual-rollout-blocked-at-max-versions-ineligible-for-deletion","WorkerID":"integration-tests","BuildID":"v0-d8d8","WorkerType":"WorkflowWorker","Error":"closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR, debug data: \"graceful_stop\""}

Overall, it's virtually impossible to look at the log output and determine what is an expected error and what isn't. I think a good first step in improving these integration tests would be separate out the tests that assert expected failures (like Connection validation errors that really should be only unit tests, not integration tests).

Having failure-expecting scenarios in completely independent integration tests would make log-grepping much easier for the non-failure-expecting scenarios.