Hmbown / Hmbown/Codewhale

Parallel catalog-filter test run hangs (pre-existing, cross-module)

Open
#6,295 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
41k
Forks
3.6k
Avg merge
13h 59m
Merged PRs (30d)
299

Description

## What

`cargo test -p codewhale-tui --lib --locked catalog` (default parallel threads)
hangs deterministically on this machine: the test binary sits at ~1.5s CPU
after 6-10 minutes, and libtest's stall warnings name `runtime_api::tests::*`
victims (`output_cap_catalog_reports_exact_transport_support`,
`provider_catalog_and_switch_preserve_each_listed_route_identity`,
`provider_catalog_keeps_official_deepseek_facts_but_not_custom_proxy_claims`,
`provider_catalog_preserves_named_custom_identity_for_new_threads`).

## Evidence it is pre-existing

- Reproduced 3x with #6289 slice-2 changes in the tree (two full-filter
runs, one shard excluding all touched modules).
- Reproduced 1x on a stashed baseline tree (binary relinked after the
stash, same hang signature). Slice 2's production changes are pure
functions; nothing in them can block.
- Single-threaded (`--test-threads=1`) the same filter completes: 246
passed + 1 seed-fallback failure that slice 2 then fixed.
- `runtime_api::tests::` alone in parallel: 228 passed in 15s. The stall is
a cross-module interaction under the `catalog` filter, not one bad test.

## Suspects (unproven)

- Local-only interference: live Codewhale processes (pet serve, app-server)
were listening during these runs, and the runs did not use
`scripts/with-hermetic-test-home.sh` the way CI does.
- A fixed-port `spawn_test_server` collision under this particular filter's
parallelism (e.g. the `127.0.0.1:18190` lm-studio fixture).
- A genuine lock-ordering deadlock between global catalog locks held across
awaits in different modules' tests.

## Acceptance

- [ ] Reproduce (or fail to) under `with-hermetic-test-home.sh` with no live Codewhale processes
- [ ] Identify the wedged test pair via shard bisection or `lldb` backtraces of the stuck binary
- [ ] Fix the ordering/port/global-state interaction so the parallel filter completes

Contributor guide

Open the contributing guide

Research direction

Start by running `cargo test -p codewhale-tui --lib --locked catalog` and repeat it with `scripts/with-hermetic-test-home.sh`, ensuring no live Codewhale processes are running. Use shard bisection or `lldb` backtraces on the stuck binary to identify the wedged test pair, then inspect the fixed-port fixture and global catalog-lock interactions. Done means the parallel catalog filter completes reliably under the hermetic setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.