aio-libs / aio-libs/aiobotocore
CI: per-test time has roughly tripled across #1667 and #1686
- Lenguaje dominante
- Python
- Estrellas
- 1.4k
- Forks
- 209
- Merge medio
- 4 d 15 h
- PR fusionados (30 d)
- 16
Descripción
The CI test step has gone from 128s to 281s over two changes, while running
*fewer* tests. Per-test cost has roughly tripled.
All three numbers below are the same job — `Test Python 3.14 on ubuntu-24.04 httpx`
— with coverage enabled in every case, so they are directly comparable:
| commit | tests executed | `Run unittests` step | job total | per test |
|-|-|-|-|-|
| `67579a1` (before #1667) | 2395 | **128s** | 141s | 0.053s |
| `7788038` (#1667, trio port) | 1687 | **161s** | 179s | 0.095s |
| `791296c` (current main, after #1686) | 1657 | **281s** | 300s | 0.170s |
So there are two separate regressions, each roughly doubling per-test cost:
#1667 took it 0.053s → 0.095s, and #1686 took it 0.095s → 0.170s.
### Not the cause
- **Setup/install.** On current main the non-test steps total 3-4s (`Install test
dependencies` 3s, `Set up job` 2s). The `.venv` cache is hitting. The entire
regression is inside the pytest run.
- **Coverage configuration.** Coverage is on for this job at all three commits.
#1686 also unpinned coverage from `==7.2.7`, which restored the C tracer on
3.12+ — that should have made this *faster*, not slower.
- **Test count.** It went down, not up.
- **pytest's unraisable plugin.** #1686 promoted `PytestUnraisableExceptionWarning`
to an error, which looked like a candidate since that plugin can force GC — but
`gc_collect_harder()` is only called from `cleanup()` at session end, not per
test, so it cannot account for a uniform per-test cost.
- **The machine.** A full local run of current main is ~42s wall on a developer
laptop for the same 1657 tests, so the suite is not inherently slow.
### Worth investigating
For #1667:
- The suite now generates 2576 items and deselects 889 of them. Tests that use the
shipped aiohttp default get parametrized over `asyncio`/`trio` and then have the
`[trio]` half deselected, because aiohttp is asyncio-only. That is wasted
collection, and it also makes the deselect count too noisy to notice a real
problem.
- The tests that remain each do more work: the `anyio` indirection and the trio
scheduler are on paths that previously went straight to asyncio.
For #1686, candidates not yet ruled out:
- `--dist worksteal` replaced the default `load`. It was measured as a win on the
tail of the suite locally, but has not been isolated on a 4-worker CI runner.
- The `isolate_aws_environment` autouse fixture calls `tmp_path_factory.mktemp()`
for every test, so the run now creates ~1657 directories it did not before.
- The shared `config` fixture moved from `read_timeout`/`connect_timeout` of 5s to
60s with retries disabled. Nothing should hit those timeouts on a passing run,
but a slow moto response that used to fail fast now blocks far longer.
### Suggested approach
Bisect inside the pytest invocation rather than across commits: run current main in
CI with `--dist load`, then with the autouse fixture reduced to session scope, then
with the 5s timeouts restored, and compare the `Run unittests` step each time. Each
is a one-line change and the step duration is a clean signal.
Related: #1687.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.