debops / debops/debops

[pki] realm-refresh-keys certbot deploy hook silently no-ops when realm name differs from certbot lineage name

Open
#2,696 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jinja
Stars
1.4k
Forks
379
Avg merge
4d 18h
Merged PRs (30d)
8

Description

Summary

The certbot deploy hook installed by the debops.pki role
(ansible/roles/pki/files/etc/letsencrypt/renewal-hooks/deploy/pki-realm-refresh-keys)
assumes that the certbot lineage name (basename "${RENEWED_LINEAGE}")
matches the PKI realm directory name under /etc/pki/realms/. When the two
differ, the hook exits with status 0 having performed no work, leaving the
realm's chain files (public/cert_intermediate.pem,
public/cert_intermediate_dhparam.pem, private/key_chain.pem,
private/key_chain_dhparam.pem) stale after every renewal.

The mapping assumption contradicts a configuration pattern that the official
debops.pki documentation explicitly recommends, so any user who follows that
documentation will hit this defect on first renewal.

Affected versions

  • Confirmed on debops.debops collection 3.2.5.
  • master (HEAD at the time of writing) ships the byte-identical script;
    sha256sum of the file is the same at the v3.2.5 tag and at master.
  • The 3.3.0 changelog touches the pki role but does not address this
    mapping.

Offending logic

realm_domain="$(basename "${RENEWED_LINEAGE}")"

if [ -n "${realm_domain}" ] && [ -d "/etc/pki/realms/${realm_domain}" ] ; then
    cd "/etc/pki/realms/${realm_domain}" > /dev/null

    # ... rebuild cert_intermediate*.pem and key_chain*.pem ...
fi

The script derives the realm directory exclusively from
basename "${RENEWED_LINEAGE}". RENEWED_LINEAGE is set by certbot to
/etc/letsencrypt/live/<lineage-name>, where <lineage-name> is the first
-d argument certbot was invoked with (unless overridden via --cert-name).
The script provides no fallback path when the lineage name does not match a
realm directory.

Why this contradicts the documentation

The "Certificate for subdomains, excluding the apex domain" example in the
debops.pki ACME integration docs
(https://docs.debops.org/en/master/ansible/roles/pki/acme-integration.html#example-certificate-for-subdomains-excluding-the-apex-domain)
recommends configuring a realm whose name: is the Subject CN while
acme_domains: contains a different domain that ends up as the certbot
SAN/lineage:

pki_realms:
  - name: 'logs.example.com'
    acme: True
    acme_default_subdomains: []
    acme_domains: [ 'mon.example.com' ]

With this configuration:

  • The realm directory created by pki-realm is
    /etc/pki/realms/logs.example.com/.
  • The first -d certbot argument is mon.example.com, so certbot's lineage
    is /etc/letsencrypt/live/mon.example.com/ and at deploy-hook time
    basename "${RENEWED_LINEAGE}" returns mon.example.com.
  • The hook tests [ -d "/etc/pki/realms/mon.example.com" ], which is false,
    so the if body is skipped and the hook exits 0.

The realm's cert_intermediate*.pem and key_chain*.pem files therefore stay
pinned to the previous renewal's keypair until something else (for example the
periodic pki-realm run, the next Ansible converge, or a manual
pki-realm run --name=<realm>) rebuilds them. Any service that loads
default.crt / default.key (or key_chain*.pem) on (re)start in that
window sees a Subject Public Key Info mismatch between the public chain and
the private key, which is exactly the failure mode the hook was added in
PR #2137 to prevent.

Minimal reproduction

  1. Deploy a host with the debops.pki role.

  2. Configure the realm as documented above:

    pki_realms:
      - name: 'logs.example.com'
        acme: True
        acme_default_subdomains: []
        acme_domains: [ 'mon.example.com' ]
    
  3. Let pki-realm issue the certificate via certbot. The realm directory is
    /etc/pki/realms/logs.example.com/; certbot's lineage is
    /etc/letsencrypt/live/mon.example.com/.

  4. Force a renewal:

    certbot renew --cert-name mon.example.com --force-renewal
    
  5. Observe that journalctl --since "1 minute ago" shows no log message from
    the pki-realm-refresh-keys hook for logs.example.com, and that
    /etc/pki/realms/logs.example.com/private/key_chain.pem is not refreshed
    (its modification time predates the renewal, while
    /etc/letsencrypt/live/mon.example.com/privkey.pem is new).

  6. openssl x509 -modulus -noout -in /etc/pki/realms/logs.example.com/public/cert_intermediate.pem | openssl md5
    vs.
    openssl rsa -modulus -noout -in /etc/pki/realms/logs.example.com/private/key.pem | openssl md5
    will report different moduli until the next pki-realm run.

How this was found

By comparing the upstream documentation's recommended "subdomains excluding
the apex" pki_realms example with the deploy-hook source: the hook is
keyed only on basename "${RENEWED_LINEAGE}", while the documented example
guarantees that this basename will not equal the realm directory name.

Proposed fix

pki-realm already writes the configured acme_domains to the realm's own
config file (/etc/pki/realms/<realm>/config/realm.conf) as a bash-sourceable
line of the form:

config['acme_domains']='dom1/dom2/...'

certbot exports a second variable to deploy hooks alongside RENEWED_LINEAGE:
RENEWED_DOMAINS, a space-separated list of every SAN in the renewed
certificate. The hook can iterate /etc/pki/realms/*/config/realm.conf,
source each one (in a subshell), split config[acme_domains] on /, and
rebuild any realm whose acme_domains intersects with RENEWED_DOMAINS.
This removes the basename assumption entirely and makes the hook correct for
both the apex-included and apex-excluded variants documented upstream.

Additionally, the hook should emit a single
logger -t pki-realm-refresh-keys line when it finds no matching realm, so
the "silent no-op" failure mode becomes visible in journalctl.

Sketch:

#!/usr/bin/env bash
set -o nounset -o pipefail -o errexit

[ -d "/etc/letsencrypt/live" ] || exit 0

renewed_domains=" ${RENEWED_DOMAINS:-} "
matched=0

shopt -s nullglob
for conf in /etc/pki/realms/*/config/realm.conf ; do
    declare -A config=()
    # shellcheck disable=SC1090
    . "${conf}"

    realm_dir="$(dirname "$(dirname "${conf}")")"
    realm_acme_domains="${config[acme_domains]:-}"
    [ -n "${realm_acme_domains}" ] || continue

    IFS='/' read -ra acme_list <<<"${realm_acme_domains}"
    for d in "${acme_list[@]}" ; do
        [ -n "${d}" ] || continue
        case "${renewed_domains}" in
            *" ${d} "*)
                refresh_realm "${realm_dir}"   # body of the current `if`
                matched=1
                break
                ;;
        esac
    done
    unset config
done

if [ "${matched}" -eq 0 ] ; then
    logger -t pki-realm-refresh-keys \
        "No PKI realm matched RENEWED_DOMAINS='${RENEWED_DOMAINS:-}' / RENEWED_LINEAGE='${RENEWED_LINEAGE:-}'"
fi

If preserving the current single-realm fast path is desirable, the
basename "${RENEWED_LINEAGE}" branch can be kept and the RENEWED_DOMAINS
loop used only as a fallback when that directory does not exist.

Why this is a doc/code inconsistency

The documented "subdomains excluding the apex" example and the deploy hook
are mutually inconsistent: following the docs produces realms that the hook
cannot find, while the hook only works for realms whose name: matches the
first entry of acme_domains (i.e. the apex-included variant). Either the
docs need a warning that the realm name: must equal the certbot lineage
name, or the hook needs to stop relying on that equality. Fixing the hook
is the safer change because the docs' example is a legitimate and useful
configuration that has no other obstacle.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ansible/roles/pki/files/etc/letsencrypt/renewal-hooks/deploy/pki-realm-refresh-keys and compare its realm lookup with /etc/pki/realms/*/config/realm.conf. Reproduce the documented logs.example.com and mon.example.com configuration, then run certbot renew --cert-name mon.example.com --force-renewal. Done means matching realms refresh their chain files and an unmatched renewal emits the pki-realm-refresh-keys logger message.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, bash
Domain
security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.