apache / apache/arrow

[C++][CI] arrow-gcsfs-test reports "failed to listen" when the testbench is just slow to start

Open
#51,011 0 comments 0 reactions 1 assignee Claimed by @jonasdedden View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

`arrow-gcsfs-test` fails on CI with every test reporting:

```
Could not start GCS emulator 'storage-testbench' (failed to listen)
```

Seen on `AMD64 macOS 15-intel C++`, all three `--repeat until-pass:3` attempts. The testbench installed fine and `type storage-testbench` succeeded.

`GcsTestbench` in `cpp/src/arrow/filesystem/gcsfs_test.cc` gives startup a 10 second budget:

```cpp
auto ready_timeout = std::chrono::seconds(10);
end = now() + ready_timeout;
while (server_process->IsRunning() && now() < end) {
auto client = gcs::Client(...
.set(gcs::LimitedTimeRetryPolicy(ready_timeout).clone()));
auto metadata = client.GetBucketMetadata("nonexistent");
...
}
```

The per-attempt retry budget equals the overall budget, so the loop gets one attempt, not a poll.

That budget is tight because of how the testbench serves. Its entry point ends with `run_simple(..., use_reloader=True)`, and Werkzeug binds the listening socket in the parent, then spawns a reloader child that re-imports grpcio, protobuf and flask before answering anything. Measured with werkzeug 3.0.4 and a 2.0s stand-in for those imports:

```
use_reloader=1: TCP accept at 2.10s first HTTP response at 4.17s
use_reloader=0: TCP accept at 2.07s first HTTP response at 2.07s
```

So the port accepts connections immediately, the first response arrives after a second full import, and the message is wrong: the testbench did listen.

`server_process->IgnoreStderr()` at line 80 discards Werkzeug's output, so there is nothing in the CI log to tell a slow start from a real crash.

Suggested: bound each attempt separately from the overall budget, raise the overall budget, keep stderr, and fix the message.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.