hardbyte / hardbyte/awa

CI flake: Python suite fixture pool-timeouts in late test_start_config tests (~15 min into the run)

Open
#420 2 comments 0 reactions 0 assignees View on GitHub
operational
Dominant language
Rust
Stars
32
Forks
5
Avg merge
15h 21m
Merged PRs (30d)
21

Description

Recurring pattern, three sightings on 2026-07-11 alone (PR #409 run 29144621874; PR #410 runs at 57c7394-era and 64c58ce):

- The Python suite passes 307-308 tests, then 1-2 tests **at the very end of the run** (~15-17 minutes in, always in `tests/test_start_config.py` so far: `test_default_start_on_fresh_install_auto_finalizes_to_queue_storage`, `test_tuple_form_backward_compat`, `test_partitioned_queue_configs_expand_and_dispatch`) ERROR at fixture setup with `_awa.DatabaseError: pool timed out while waiting for an open connection`.
- Reruns pass. The affected branches touch no pool code.

The end-of-suite + fixture-setup signature suggests connection accumulation across the session (leaked pools from earlier tests exhausting the service container's max_connections, or client pools not fully closed — note the #408 class of runtime-teardown pool behavior in the bridge). test_start_config tests create fresh clients with their own pools, which makes them the first victims of an exhausted server.

Diagnosis ideas: log `pg_stat_activity` count from a fixture at suite start/end; or set a lower `max_connections` locally to reproduce faster; or audit which tests create AsyncClients without closing.

Not release-blocking (rerun-clearable), but it costs a rerun on most full-ci passes now — same operational-cost class as #399.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with tests/test_start_config.py and reproduce the late fixture-setup failure using a lower local max_connections setting. Inspect suite fixtures and tests that create AsyncClients, then use pg_stat_activity at suite start and end to identify accumulated connections. Done means repeated full Python-suite runs complete without pool timeouts and the affected tests remain reliable without reruns.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, rust
Domain
ci-cd, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.