aio-libs / aio-libs/aiobotocore
CI test-suite performance: moto server scope, runner class, matrix scoping
- Linguagem predominante
- Python
- Estrelas
- 1.4k
- Forks
- 209
- Merge médio
- 4d 15h
- PRs com merge (30d)
- 16
Descrição
## Summary
The `moto_server` fixture is function-scoped, so every moto-backed test starts and
stops its own `ThreadedMotoServer`. Teardown costs a fixed ~0.5s, and the isolation
it appears to buy is an illusion. Making it session-scoped is worth roughly a **5x**
speedup on moto-heavy files, but two things have to be fixed first — this issue
records the measurements and the blockers.
## The cost
```
first start (cold): 72.8 ms
warm start: 5.8 ms
warm stop: 501.1 ms <-- per test
```
The 501ms is not moto's code. `ThreadedMotoServer.stop()` calls
`self._server.shutdown()`, and werkzeug's `serve_forever` polls on a **0.5s
interval**, so shutdown blocks until the next tick.
Measured on `tests/test_sqs.py tests/test_sns.py tests/test_dynamodb.py
tests/test_ec2.py` (32 tests):
| `moto_server` scope | wall |
|-|-|
| function (today) | 58.1s |
| session | 11.5s |
The subset runs at ~3% CPU, so this is waiting, not compute — a profiler
(cProfile/yappi) mostly shows idle here; the timing above is the useful instrument.
## Why per-test servers isolate nothing
moto's backends are process-global singletons. A fresh `ThreadedMotoServer` on a new
port still sees resources created by earlier tests in the same xdist worker. This is
already observable today: it is exactly how #1686's `test_waiter` failure happened —
a stack leaked by a timed-out attempt collided with the retry and with the other HTTP
backend parametrization, across two different server instances.
So the per-test server costs 0.5s and provides no isolation. Isolation comes from
unique resource names (`random_name()`), which most fixtures already use.
## Blocker 1 — tests that assert on listings
Unique names are necessary but not sufficient: a test that *lists* a resource type
sees everything the worker created. Audit of current listing assertions:
- `test_sqs.py::test_list_queues` — `assert sqs_queue_url in response['QueueUrls']`, membership, safe
- `test_dynamodb.py:99` — `assert table_name not in response['TableNames']`, safe
- `test_basic_s3.py` `list_buckets` tests — assert response *structure* only ("Can't
really assume anything about whether or not they have buckets"), safe
- `test_basic_s3.py` `list_objects` — always scoped to a per-test `Bucket=`, safe
- **`test_sns.py:38` — `arn1 = response['Topics'][0]['TopicArn']` then asserts it equals
the topic this test created. Not safe under a shared server.** Needs to select its own
ARN out of the list rather than index `[0]`.
## Blocker 2 — the suite hangs at teardown
With the fixture switched to `scope='session'`, the full suite hangs reproducibly at
~98% under `-n auto`. `SIGABRT` on the controller (pytest enables faulthandler, so no
root needed):
```
Current thread (controller):
xdist/dsession.py:139 in loop_once -> queue.get() -> threading wait
Receiver threads:
execnet/gateway_base.py:534 in read # blocked reading, not EOF
```
The controller is waiting on worker events and the execnet channels are still open, so
a worker is alive and silent. `faulthandler_timeout=120` never fires, which means the
stall is **outside any test** — i.e. session-fixture teardown. The prime suspect is
`ThreadedMotoServer.stop()`'s `self._thread.join()`, which has no timeout; it only runs
once per session under this change.
Needs a dump of a hung *worker* (they run as `python -c 'import execnet...'`, so they
don't match `pgrep pytest`) to confirm before fixing.
## ~~Related cleanup in `tests/mock_server.py`~~ — DONE in #1686
Everything in this section shipped in #1686 (9c6797d). Leaving the detail below for
context, but it is no longer work to pick up.
`AIOServer` turned out to be actively breaking CI rather than merely untidy: it leaked
its child process on a failed start, because `__aexit__` only runs when `__aenter__`
*returns*, and never reaped on the success path either (`terminate()` without `join()`).
A 3.12 job failed 16 tests with `unable to start and connect to aiohttp server` and left
~34 orphan processes for 4 workers -- leaked servers starving the next ones on a 4-vCPU
runner, a feedback loop ending in a job timeout. Verified: 5 failed starts leave 5 live
children before the fix, 0 after.
It is now fully event-driven, as originally sketched here: the child binds port 0 and
sends the bound URL back over a `Pipe` (removing the reserve-then-release port race and
`get_free_tcp_port` with it), that message doubles as the readiness signal (removing the
30x0.5s poll loop), and shutdown is an `Event` the child awaits, with `_shutdown`
escalating join -> terminate -> kill. `stream_handler` waits on that same event, so it
still outlasts the client read timeout but no longer holds teardown open.
Measured on the 28 AIOServer tests: 25.87s -> 13.78s, with both timeout tests now at
their intrinsic floor rather than the server's:
| test | before (CI) | after |
|-|-|-|
| `test_connector_timeout` | 12.98-15.43s | 3.37s (its own `move_on_after(3)`) |
| `test_connector_timeout2` | 10.00-11.27s | 1.39s (its own `read_timeout=1`) |
| `test_useragent` x6 | ~10s each | <1s each |
Still open from the original list: nothing. The `pytest.fail`-in-the-child and dead
`return`/`raise` lines went with the rewrite.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.