aio-libs / aio-libs/aiobotocore

CI: per-test time has roughly tripled across #1667 and #1686

Abierto
#1,689 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
1.4k
Forks
209
Merge medio
4 d 15 h
PR fusionados (30 d)
16

Descripción

The CI test step has gone from 128s to 281s over two changes, while running
*fewer* tests. Per-test cost has roughly tripled.

All three numbers below are the same job — `Test Python 3.14 on ubuntu-24.04 httpx`
— with coverage enabled in every case, so they are directly comparable:

| commit | tests executed | `Run unittests` step | job total | per test |
|-|-|-|-|-|
| `67579a1` (before #1667) | 2395 | **128s** | 141s | 0.053s |
| `7788038` (#1667, trio port) | 1687 | **161s** | 179s | 0.095s |
| `791296c` (current main, after #1686) | 1657 | **281s** | 300s | 0.170s |

So there are two separate regressions, each roughly doubling per-test cost:
#1667 took it 0.053s → 0.095s, and #1686 took it 0.095s → 0.170s.

### Not the cause

- **Setup/install.** On current main the non-test steps total 3-4s (`Install test
dependencies` 3s, `Set up job` 2s). The `.venv` cache is hitting. The entire
regression is inside the pytest run.
- **Coverage configuration.** Coverage is on for this job at all three commits.
#1686 also unpinned coverage from `==7.2.7`, which restored the C tracer on
3.12+ — that should have made this *faster*, not slower.
- **Test count.** It went down, not up.
- **pytest's unraisable plugin.** #1686 promoted `PytestUnraisableExceptionWarning`
to an error, which looked like a candidate since that plugin can force GC — but
`gc_collect_harder()` is only called from `cleanup()` at session end, not per
test, so it cannot account for a uniform per-test cost.
- **The machine.** A full local run of current main is ~42s wall on a developer
laptop for the same 1657 tests, so the suite is not inherently slow.

### Worth investigating

For #1667:

- The suite now generates 2576 items and deselects 889 of them. Tests that use the
shipped aiohttp default get parametrized over `asyncio`/`trio` and then have the
`[trio]` half deselected, because aiohttp is asyncio-only. That is wasted
collection, and it also makes the deselect count too noisy to notice a real
problem.
- The tests that remain each do more work: the `anyio` indirection and the trio
scheduler are on paths that previously went straight to asyncio.

For #1686, candidates not yet ruled out:

- `--dist worksteal` replaced the default `load`. It was measured as a win on the
tail of the suite locally, but has not been isolated on a 4-worker CI runner.
- The `isolate_aws_environment` autouse fixture calls `tmp_path_factory.mktemp()`
for every test, so the run now creates ~1657 directories it did not before.
- The shared `config` fixture moved from `read_timeout`/`connect_timeout` of 5s to
60s with retries disabled. Nothing should hit those timeouts on a passing run,
but a slow moto response that used to fail fast now blocks far longer.

### Suggested approach

Bisect inside the pytest invocation rather than across commits: run current main in
CI with `--dist load`, then with the autouse fixture reduced to session scope, then
with the 5s timeouts restored, and compare the `Run unittests` step each time. Each
is a one-line change and the step duration is a clean signal.

Related: #1687.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.