CI stability audit: unpinned/floating dependencies that can flip CI from green to red

Summary

Audit (verified line-by-line against 4f922a646b3b0efa004bc20fcf71cd75491d3593) of everything in this repo whose resolved behavior can change while the git SHA under test stays exactly the same. Any of these can turn a previously-green commit red with zero code changes on our side.

Verification confirmed the lockfile story is genuinely solid: poetry.lock (root + enterprise/), frontend/package-lock.json, and openhands-ui/bun.lock are all real, committed, and correctly enforced by every CI/Docker install path (poetry install, poetry export --only main → pip install -r requirements.txt, npm ci, bun install --frozen-lockfile). The risk below is scoped to: (a) install paths that bypass those locks, (b) a metadata inconsistency inside pyproject.toml itself, and (c) mutable Action/script references.

Package metadata: root pyproject.toml declares dependencies twice, and they've drifted

The root pyproject.toml has both a PEP 621 [project.dependencies] array (read by pip/uv/anyone installing the published wheel) and a classic [tool.poetry.dependencies] table (what Poetry's resolver / poetry.lock actually uses) — and for several packages, only the classic table has an upper bound:

• Genuinely unbounded in both declarations — anthropic[vertex] L25, boto3 L31, fastapi L37, google-cloud-aiplatform L43, google-genai L44, json-repair L51, mcp (>=1.25, L57 — no Poetry-table override, so Poetry resolves straight off this uncapped PEP 621 entry), types-toml L104, uvicorn L106 (duplicated as bare "*" in the poetry table at L166/172/174/175/177/234/235/236, and in [dependency-groups]: build, types-setuptools, pandas, jupyterlab). libtmux (>=0.46.2, L46 / Poetry table L199) and joserfc (>=1.0.0, L50 / Poetry table L254) are likewise uncapped in both places — not a PEP621-vs-Poetry drift case, just plain unbounded everywhere.
• Capped only in the Poetry table + lockfile, uncapped in the PEP 621 array — meaning a wheel/sdist consumer (pip install openhands-ai, a downstream repo) sees a looser constraint than what CI/Docker actually run: google-api-python-client (PEP621 >=2.164 L40 vs Poetry ^2.164.0 L167), pathspec (L67 vs >=0.12.1,<1.1.0 L187), rapidfuzz (L91 vs ^3.9.0 L192), tree-sitter-language-pack (L103 vs ^0.7.3 L193), pygithub (L77 vs ^2.5.0 L200), python-json-logger (L84 vs >=3.2.1,<5.0.0 L206), kubernetes (L53 vs >=33.1,<36.0 L237), sqlalchemy (L96 vs ^2.0.40 L255), asyncpg (L27 vs >=0.30,<0.32 L257), lmnr (L56 vs ^0.7.20 L259).
• Only fastmcp (>=3.2,<4, L38) has a real upper bound in the PEP 621 array itself.

Fix: keep one source of truth for version constraints (ideally the PEP 621 array, generated from or validated against the Poetry table) so the published wheel metadata can't silently diverge from what's actually tested.

CI: unlocked/unpinned installs

• .github/workflows/py-tests.yml:46,81 — pipx install poetry, unpinned, in two jobs; L55-56 — unpinned pip install pytest-xdist pytest-rerunfailures, installed outside poetry.lock.
• .github/workflows/enterprise-check-migrations.yml:110 — pipx install poetry, unpinned.
• .openhands/setup.sh:6 — pip install pre-commit, unpinned.
• Makefile:160 — pip install chroma-hnswlib (macOS branch); not even declared in pyproject.toml. L171 — pip install playwright (Manjaro branch) can silently override the 1.58.0 pinned in pyproject.toml.
• Makefile:193 — install-frontend-dependencies uses plain npm install instead of npm ci (every CI workflow correctly uses npm ci; this is a local-dev-only unlocked path).

Docker: unpinned/mutable installs

• containers/app/Dockerfile:140 — pip install poetry poetry-plugin-export, unpinned, in the production build stage.
• containers/app/Dockerfile:126 and containers/dev/Dockerfile:65 — both pipe NodeSource's mutable setup_24.x/setup_22.x script straight into bash.
• containers/dev/Dockerfile:69 — curl install.python-poetry.org | python3.12 -, no --version argument.
• containers/dev/Dockerfile:17-22 — docker-ce/docker-ce-cli/containerd.io/docker-buildx-plugin/docker-compose-plugin installed unpinned from Docker's apt repo.
• containers/dev/Dockerfile:49-53 — GitHub CLI installed unpinned from cli.github.com's apt repo.
• containers/dev/Dockerfile:59-62 — deadsnakes PPA + unpinned apt-get install python3.12.

Pre-commit

• dev_config/python/.pre-commit-config.yaml and enterprise/dev_config/python/.pre-commit-config.yaml — hook repos pinned by tag (v5.0.0, v2.5.1, v0.24.1, v0.12.5, v1.15.0), not commit SHA.
• Same files' mypy hook additional_dependencies (types-requests, types-setuptools, types-pyyaml, types-toml, types-docker, types-Markdown, pydantic, lxml, sqlalchemy>=2.0, types-redis) are mostly unpinned and installed via plain pip into an isolated hook venv, outside poetry.lock entirely. lint.yml/lint-fix.yml never cache ~/.cache/pre-commit, so a clean CI run re-resolves "latest" for each unpinned entry every time — a new PyPI release of pydantic/lxml/any types-* stub can flip mypy from green to red on an unchanged commit.

Mutable GitHub Actions

• bump-chart.yml:21 — OpenHands/OpenHands-Cloud/...@main.
• pr.yml:14, release.yml:24 — OpenHands/release-actions/...@main.
• qa-changes-by-openhands.yml:32 — @main.
• issue-opened.yml:77,330 — issue-duplicate-checker@v0.4.1, a tag not a SHA.
• Official actions/* (checkout@v6, setup-python@v6, setup-node@v6, upload-artifact@v7, download-artifact@v8, github-script@v9) float on major tags throughout.
• Confirmed as accurate: every other third-party action across all 18 workflow files + 2 composite actions (docker/*, peter-evans/*, py-cov-action/*, oven-sh/setup-bun, actions/create-github-app-token) is already correctly SHA-pinned — the items above are the only stragglers.

Unpinned base images

• node:25.9-trixie-slim — containers/app/Dockerfile:7
• python:3.13.7-slim-trixie — containers/app/Dockerfile:17
• ubuntu:26.04 — containers/dev/Dockerfile:4
• postgres:16 — enterprise-check-migrations.yml:88
• ubuntu:22.04 — kind/manifests/deployment.yaml:18 (local kind-cluster manifest only, not in the CI build path — low priority).
• image: openhands:latest — docker-compose.yml:9 (local-dev compose file only, doesn't gate CI — low priority).

Runner image drift

• runs-on: ubuntu-latest used in several workflows (check-package-versions.yml, check-version-consistency.yml, enterprise-check-migrations.yml, py-tests.yml, issue-opened.yml, pr-artifacts.yml) — GitHub's rolling runner image can change preinstalled tool/library versions under an unchanged commit.

Suggested priority

1. Reconcile the PEP 621 vs Poetry-table dependency drift in root pyproject.toml — this is the one finding where the committed constraint and the published constraint genuinely disagree.
2. Pin the pre-commit hooks (SHA) and the mypy hook's additional_dependencies, and cache ~/.cache/pre-commit in lint.yml/lint-fix.yml.
3. Pin the 4 mutable Action refs and SHA-pin the official actions/* steps.
4. Replace the three curl | sh-style installers (uv-less here, but NodeSource ×2 + install.python-poetry.org) with pinned, checksum-verified installs.
5. Digest-pin the Docker base images.

Start with the priority list in root pyproject.toml, then inspect the referenced workflows, pre-commit configurations, Makefile, and Dockerfiles. Compare the PEP 621 and Poetry constraints first, followed by the listed unlocked installs, mutable action references, and image tags. Done means the identified sources are reconciled or pinned without leaving the audit items unresolved.