AlmaLinux / AlmaLinux/build-system
Investigate intermittent RPM sign failures — gpg-agent passphrase cache appears to expire mid-run
- Ngôn ngữ chính
- Không có dữ liệu ngôn ngữ
- Star
- 32
- Fork
- 11
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Summary
Sign node intermittently fails RPM signing with `PackageSignError: RPM sign failed with N exit code`. Observed on AlmaLinux builds [66904](https://build.almalinux.org/build/66904), [66905](https://build.almalinux.org/build/66905), and [66908](https://build.almalinux.org/build/66908) on 2026-07-15. The only known workaround is restarting the sign node service and re-entering passphrases.
## Symptoms
Top-level error reported to the web server:
```
sign_node.package_sign.PackageSignError: RPM sign failed with 1 exit code.
Traceback: NoneType: None
```
(also seen with exit code `9`)
The underlying `rpmsign` output within a **single batch of packages** degrades through a sequence of distinct GPG errors for the same key:
```
gpg: signing failed: General error
gpg: signing failed: Missing item in object
gpg: signing failed: No secret key
error: gpg exec failed (2)
```
Early packages in the batch sign fine (`Enter passphrase:` handled), then later packages in the same run start failing. This pattern points at **gpg-agent / passphrase-cache state going bad mid-run**, not a per-package defect.
## Impact
- Signing tasks fail and must be manually retried.
- Only recovery so far is `systemctl restart` of the sign node + operator re-entering all key passphrases.
## Where in the code
- `sign_node/signer.py:~441` — RPMs signed in batches of 50, passphrase fed to `rpmsign` via `pexpect` (`events={"Enter passphrase:.*": ...}`).
- `sign_node/package_sign.py` — `sign_rpm_package()` / `gpg_sign_locks()`; raises `PackageSignError` on non-zero `rpmsign` exit.
- Related recent work: `Rework filesystem locks for gpg commands` (#181), `fetch GPG passphrases from Bitwarden vault` (#182), `Add filesystem locks for interacting with YubiKeys` (#175).
## Hypotheses to investigate
1. **gpg-agent passphrase cache expiry** — `default-cache-ttl` / `max-cache-ttl` in `gpg-agent.conf` may be shorter than a long signing run, so the cache empties and later `rpmsign` invocations can't unlock the key. The `pexpect` passphrase-feeding relies on `Enter passphrase:` appearing; if gpg-agent behaves differently after cache expiry the events matcher may not re-supply it.
2. **gpg-agent restart/reload racing an in-flight sign** — the shared/exclusive lock rework (#181) was aimed at this; verify the locks actually prevent a reload during a non-Yubikey sign batch.
3. **Yubikey vs software key path** — confirm which key path (`yubikey_keyids`) the failing builds used, since the reload-after-sign only fires for Yubikey keys.
## Suggested next steps
- Reproduce with a long/large signing task and observe gpg-agent cache behavior; consider raising `max-cache-ttl` or refreshing the cache per batch.
- Improve error reporting: the emitted `Traceback: NoneType: None` comes from `traceback.format_exc()` being called with no active exception — replace it with the actual captured `rpmsign` output so future incidents are diagnosable from the payload alone.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.