AlmaLinux / AlmaLinux/build-system

Investigate intermittent RPM sign failures — gpg-agent passphrase cache appears to expire mid-run

Abierto
#542 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Sin datos de lenguaje
Estrellas
32
Forks
11
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Summary
Sign node intermittently fails RPM signing with `PackageSignError: RPM sign failed with N exit code`. Observed on AlmaLinux builds [66904](https://build.almalinux.org/build/66904), [66905](https://build.almalinux.org/build/66905), and [66908](https://build.almalinux.org/build/66908) on 2026-07-15. The only known workaround is restarting the sign node service and re-entering passphrases.

## Symptoms
Top-level error reported to the web server:
```
sign_node.package_sign.PackageSignError: RPM sign failed with 1 exit code.
Traceback: NoneType: None
```
(also seen with exit code `9`)

The underlying `rpmsign` output within a **single batch of packages** degrades through a sequence of distinct GPG errors for the same key:
```
gpg: signing failed: General error
gpg: signing failed: Missing item in object
gpg: signing failed: No secret key
error: gpg exec failed (2)
```
Early packages in the batch sign fine (`Enter passphrase:` handled), then later packages in the same run start failing. This pattern points at **gpg-agent / passphrase-cache state going bad mid-run**, not a per-package defect.

## Impact
- Signing tasks fail and must be manually retried.
- Only recovery so far is `systemctl restart` of the sign node + operator re-entering all key passphrases.

## Where in the code
- `sign_node/signer.py:~441` — RPMs signed in batches of 50, passphrase fed to `rpmsign` via `pexpect` (`events={"Enter passphrase:.*": ...}`).
- `sign_node/package_sign.py` — `sign_rpm_package()` / `gpg_sign_locks()`; raises `PackageSignError` on non-zero `rpmsign` exit.
- Related recent work: `Rework filesystem locks for gpg commands` (#181), `fetch GPG passphrases from Bitwarden vault` (#182), `Add filesystem locks for interacting with YubiKeys` (#175).

## Hypotheses to investigate
1. **gpg-agent passphrase cache expiry** — `default-cache-ttl` / `max-cache-ttl` in `gpg-agent.conf` may be shorter than a long signing run, so the cache empties and later `rpmsign` invocations can't unlock the key. The `pexpect` passphrase-feeding relies on `Enter passphrase:` appearing; if gpg-agent behaves differently after cache expiry the events matcher may not re-supply it.
2. **gpg-agent restart/reload racing an in-flight sign** — the shared/exclusive lock rework (#181) was aimed at this; verify the locks actually prevent a reload during a non-Yubikey sign batch.
3. **Yubikey vs software key path** — confirm which key path (`yubikey_keyids`) the failing builds used, since the reload-after-sign only fires for Yubikey keys.

## Suggested next steps
- Reproduce with a long/large signing task and observe gpg-agent cache behavior; consider raising `max-cache-ttl` or refreshing the cache per batch.
- Improve error reporting: the emitted `Traceback: NoneType: None` comes from `traceback.format_exc()` being called with no active exception — replace it with the actual captured `rpmsign` output so future incidents are diagnosable from the payload alone.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.