Stabilize tls_stress_test metrics queries under exhausted session caps
- Dominant language
- C++
- Stars
- 876
- Forks
- 260
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 157
Description
## Summary
`tls_stress_test` is intermittently failing in AL4 VMSS Virtual C while the
connection-cap scenario deliberately exhausts sessions and file descriptors.
After expected connection failures, the test performs an unguarded
`GET /node/metrics` through `clients[0]`. That request may be reset or receive
`503 SessionCapExhausted`, failing the whole CI job.
This has occurred on unrelated PRs and predates the PR where it was most
recently observed.
## Most recent occurrence
- PR: https://github.com/microsoft/CCF/pull/8165
- Failed job:
https://github.com/microsoft/CCF/actions/runs/32135397827/job/95705417848
- Workflow/job: `Continuous Integration AL4` / `AL4 VMSS Virtual C`
- Test: `tls_stress_test`
- Failure path:
```text
tests/connections.py:231 run_connection_caps_tests
tests/connections.py:176 create_connections_until_exhaustion
r = clients[0].get("/node/metrics")
tests/infra/clients.py:696
raise CCFIOException from exc
infra.clients.CCFIOException
```
The underlying exception was:
```text
ConnectionResetError: [Errno 104] Connection reset by peer
httpx.ReadError: [Errno 104] Connection reset by peer
```
The normal `VMSS Virtual C` job passed on the same PR commit; only the AL4 job
failed:
- Passed:
https://github.com/microsoft/CCF/actions/runs/32135397799/job/95705418136
- Failed:
https://github.com/microsoft/CCF/actions/runs/32135397827/job/95705417848
## Evidence that this is pre-existing and intermittent
The same request at `tests/connections.py:176` failed with
`infra.clients.CCFIOException` on July 26:
- Historical job:
https://github.com/microsoft/CCF/actions/runs/30214888955/job/89827281197
- Unrelated recovery PR:
https://github.com/microsoft/CCF/pull/8092
That recovery PR did not modify `tests/connections.py` or the TLS/session-cap
implementation. Its preceding AL4 run passed, this run failed, and its next 11
AL4 runs passed without a change to `tests/connections.py`.
Related failures in the same connection-cap scenario have also returned
`503 SessionCapExhausted` from metrics queries:
- https://github.com/microsoft/CCF/actions/runs/30554767740/job/90912282664
- https://github.com/microsoft/CCF/actions/runs/30549183462/job/90893078340
- https://github.com/microsoft/CCF/actions/runs/30535447937/job/90847555194
The metrics request at `tests/connections.py:176` dates from 2021. The loop
immediately before it already treats `CCFConnectionException`,
`CCFIOException`, and `RuntimeError` as expected while searching for the
session/file-descriptor limit, but the subsequent metrics request has no
equivalent handling.
## Relevant code
- `tests/connections.py`
- `get_session_metrics()`
- `run_connection_caps_tests()`
- nested `create_connections_until_exhaustion()`
- particularly the unguarded `clients[0].get("/node/metrics")`
- `tests/infra/clients.py`
- conversion of `httpx.ReadError` to `CCFIOException`
## Investigation notes
Determine which invariant is intended:
1. An already admitted session must remain usable while new sessions are
rejected. If so, the reset/503 may expose a CCF session-cap bug and the test
should retain a strict assertion.
2. A reset or temporary 503 is valid while the test is deliberately exhausting
file descriptors and session caps. If so, the test should explicitly wait
for cap state to settle and retry the metrics query using an appropriate
existing session, rather than failing on a timing-dependent request.
Avoid simply swallowing the exception. Preserve the assertions on active/peak
session metrics and ensure a persistent inability to query metrics still fails
with useful diagnostics.
Also check whether the test should use the existing `get_session_metrics()`
helper consistently rather than making a separate direct request.
## Suggested validation
1. Build Debug on Azure Linux 4, matching the failing workflow.
2. Run `tls_stress_test` repeatedly through the repository `tests.sh` wrapper.
3. Confirm the connection-cap assertions still detect real cap regressions.
4. Confirm transient reset/503 behavior no longer fails the test if it is
considered valid.
5. Run the complete AL4 bucket C job.
## Acceptance criteria
- The intended behavior of existing sessions at the soft/hard cap is explicit
in the test.
- `tls_stress_test` is stable under repeated AL4 runs.
- Persistent metrics-query failure remains actionable and fails the test.
- Session active/peak/soft-cap/hard-cap assertions are preserved.
Contributor guide
Research direction
Start in tests/connections.py at get_session_metrics(), run_connection_caps_tests(), and the nested create_connections_until_exhaustion(), then inspect tests/infra/clients.py for httpx.ReadError conversion. Run tls_stress_test repeatedly through tests.sh on Azure Linux 4 and determine the intended existing-session behavior at the caps. Done means transient metrics failures no longer make the test flaky while persistent failures and active/peak/soft-cap/hard-cap assertions remain actionable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100