[Bug] context deadline exceeded in integration tests

Describe the bug

I've created an integration test for Temporal for NixOS over at: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/temporal.nix

This creates a fresh Temporal server in a VM, and then in the testScript section uses the cli to query that server about cluster status and workflows.

For some reason, the CLI calls will randomly fail with, whilst succeeding with a simple reattempt:

waiting for success: temporal workflow list --output json
... level=ERROR msg="failed reaching server: context deadline exceeded"

I've also tried setting --connect-timeout 120s, and it happens with that too.

This has also been encountered in NixOS's CI system: https://hydra.nixos.org/build/306058770/log

Minimal Reproduction

1. Install Nix
2. git clone https://github.com/NixOS/nixpkgs # Large download
3. cd nixpkgs; nix-build -A nixosTests.temporal --check

Environment/Versions

• OS and processor: Linux
• Temporal Version: [e.g. 1.14.0?] and/or SDK version: 1.28.1
• Are you using Docker or Kubernetes or building Temporal from source? Source

Start with nixos/tests/temporal.nix and its testScript, then reproduce the failure with nix-build -A nixosTests.temporal --check. Investigate the temporal workflow list --output json and cluster-status CLI calls against the fresh VM server, using the linked Hydra log for comparison. Done means the integration test no longer intermittently fails with context deadline exceeded.