aio-libs / aio-libs/aiobotocore

CI: per-test time has roughly tripled across #1667 and #1686

Aberta
#1,689 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
1.4k
Forks
209
Merge médio
4d 15h
PRs com merge (30d)
16

Descrição

The CI test step has gone from 128s to 281s over two changes, while running
*fewer* tests. Per-test cost has roughly tripled.

All three numbers below are the same job — `Test Python 3.14 on ubuntu-24.04 httpx`
— with coverage enabled in every case, so they are directly comparable:

| commit | tests executed | `Run unittests` step | job total | per test |
|-|-|-|-|-|
| `67579a1` (before #1667) | 2395 | **128s** | 141s | 0.053s |
| `7788038` (#1667, trio port) | 1687 | **161s** | 179s | 0.095s |
| `791296c` (current main, after #1686) | 1657 | **281s** | 300s | 0.170s |

So there are two separate regressions, each roughly doubling per-test cost:
#1667 took it 0.053s → 0.095s, and #1686 took it 0.095s → 0.170s.

### Not the cause

- **Setup/install.** On current main the non-test steps total 3-4s (`Install test
dependencies` 3s, `Set up job` 2s). The `.venv` cache is hitting. The entire
regression is inside the pytest run.
- **Coverage configuration.** Coverage is on for this job at all three commits.
#1686 also unpinned coverage from `==7.2.7`, which restored the C tracer on
3.12+ — that should have made this *faster*, not slower.
- **Test count.** It went down, not up.
- **pytest's unraisable plugin.** #1686 promoted `PytestUnraisableExceptionWarning`
to an error, which looked like a candidate since that plugin can force GC — but
`gc_collect_harder()` is only called from `cleanup()` at session end, not per
test, so it cannot account for a uniform per-test cost.
- **The machine.** A full local run of current main is ~42s wall on a developer
laptop for the same 1657 tests, so the suite is not inherently slow.

### Worth investigating

For #1667:

- The suite now generates 2576 items and deselects 889 of them. Tests that use the
shipped aiohttp default get parametrized over `asyncio`/`trio` and then have the
`[trio]` half deselected, because aiohttp is asyncio-only. That is wasted
collection, and it also makes the deselect count too noisy to notice a real
problem.
- The tests that remain each do more work: the `anyio` indirection and the trio
scheduler are on paths that previously went straight to asyncio.

For #1686, candidates not yet ruled out:

- `--dist worksteal` replaced the default `load`. It was measured as a win on the
tail of the suite locally, but has not been isolated on a 4-worker CI runner.
- The `isolate_aws_environment` autouse fixture calls `tmp_path_factory.mktemp()`
for every test, so the run now creates ~1657 directories it did not before.
- The shared `config` fixture moved from `read_timeout`/`connect_timeout` of 5s to
60s with retries disabled. Nothing should hit those timeouts on a passing run,
but a slow moto response that used to fail fast now blocks far longer.

### Suggested approach

Bisect inside the pytest invocation rather than across commits: run current main in
CI with `--dist load`, then with the autouse fixture reduced to session scope, then
with the 5s timeouts restored, and compare the `Run unittests` step each time. Each
is a one-line change and the step duration is a clean signal.

Related: #1687.

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Start from the CI job `Test Python 3.14 on ubuntu-24.04 httpx` and the pytest invocation used by `Run unittests`. Compare current main with one-line trials mentioned in the issue: `--dist load`, reducing `isolate_aws_environment` fixture scope, and restoring 5s timeouts in the shared `config` fixture. Done means the regression source is isolated and the CI test step/per-test cost improves without hiding tests.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
aws, github-actions, python
Domínio
ci-cd, performance, testing
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Pouca atividade
Clareza
Razoavelmente clara
Facilidade para iniciantes
48/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.