coder / coder/internal

flake: TestPubsubCluster/ClusterAuthRequired

Open
#1,636 1 comment 0 reactions 1 assignee Claimed by @sreya View on GitHub
flake
Dominant language
No language data
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## CI failure

* **CI run:** [https://github.com/coder/coder/actions/runs/30976975074]()
* **Failing job:** [https://github.com/coder/coder/actions/runs/30976975074/job/92212913393]() (`test-go-pg (windows-2022)`)
* **Failure date:** 2026-08-05 05:15 UTC; job completed at 05:21 UTC, before the 05:29 Slack alert
* **Run attempt:** 1
* **Commit:** [`7a4ae2649eb592b961c142caf43dbe064a15a308`]()
* **Commit author:** Jake Howell

The complete 4,462-line Windows job log was available and reviewed. The macOS matrix job was canceled only after the Windows failure; this is not a rerun/cancellation artifact.

## Failure evidence

```text
=== FAIL: coderd/x/nats TestPubsubCluster/ClusterAuthRequired (0.04s)
pubsub_internal_test.go:519:
Error: Target error should be in err chain:
expected: "nats: authorization violation"
in chain: "read tcp 127.0.0.1:59696->127.0.0.1:59631: wsarecv: An existing connection was forcibly closed by the remote host."
"wsarecv: An existing connection was forcibly closed by the remote host."
"An existing connection was forcibly closed by the remote host."
Messages: route dial with wrong token must be rejected
```

The embedded NATS server started successfully and shut down cleanly. The wrong-token route connection was rejected as intended, but Windows surfaced the rejection as `WSAECONNRESET` instead of `nats.ErrAuthorization`.

## Root cause assessment

**Classification: flaky/OS-specific test assertion.**

`ClusterAuthRequired` dials the NATS **cluster route listener** with a normal nats.go client and requires the returned error to wrap `natsgo.ErrAuthorization`. That assumes the route listener always writes a client-protocol `-ERR 'Authorization Violation'` that nats.go parses before closing the socket.

On Windows, the server can close/reset the rejected route connection before the client reads a parseable protocol error. Winsock then reports `wsarecv: An existing connection was forcibly closed by the remote host`. The security behavior is correct—the wrong token was rejected—but the exact error-class assertion is not portable/stable for this route-protocol misuse.

Possible fixes:

* treat a Windows connection reset as an accepted rejection outcome in this subtest;
* assert that no route forms / the connection closes rather than requiring one exact nats.go client error;
* or test route authentication through a route-protocol-aware mechanism/server callback instead of connecting a normal NATS client to the cluster port.

## Race, panic, OOM, and resource checks

The full job log contains no Go race warning, panic, OOM, killed process, allocation failure, disk exhaustion, or file-descriptor exhaustion indicator. This is unrelated to the NATS shutdown panic tracked in coder/internal#1631.

## Assignment analysis

Ownership/history targets:

```text
git blame -L 500,530 coderd/x/nats/pubsub_internal_test.go
git log --oneline -10 --follow coderd/x/nats/pubsub_internal_test.go
```

The exact `ClusterAuthRequired` subtest and its `require.ErrorIs(..., natsgo.ErrAuthorization)` assertions were introduced together in [`167ac7b8`]() (`feat: add nats experiment`, PR [coder/coder#25703]()) by `sreya`. Later changes modified other NATS tests or only increased cluster TLS timeout; they did not alter this assertion.

Assigning `sreya` based on exact test-line ownership, not the failing CI commit author.

## Duplicate search

Searched open and closed `coder/internal` issues, including the last 30 days, for:

* `TestPubsubCluster/ClusterAuthRequired` and `TestPubsubCluster`;
* `nats: authorization violation`, wrong-token route auth, and `pubsub_internal_test.go`;
* `An existing connection was forcibly closed by the remote host` / `wsarecv`;
* NATS Windows failures, process crashes, panic/OOM/unknown failures, and race signatures.

No issue describes this test and failure mode.

Related but not duplicate:

* [https://github.com/coder/internal/issues/1631]() — NATS peer-refresh shutdown panic (`send on closed channel`), different code path/root cause.
* [https://github.com/coder/internal/issues/1617]() — `TestSubscribeError/Flush` stale cleanup, different subtest/root cause.
* [https://github.com/coder/internal/issues/1497]() — same generic Winsock wording in an AI bridge test, but a different component and assertion.

## Reproduction

```text
go test ./coderd/x/nats -run '^TestPubsubCluster/ClusterAuthRequired$' -count=100
```

Run on Windows; full-suite socket churn may increase occurrence frequency.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.