Metadata Signing concerns

Problem Description

I think there's a fundamental design flaw in the configuration and maintenance of the Metadata Signing key mechanics that Pulp provides. Some context:

• We set our keys to expire every 2 years
• Updating that key is not easy
• Documentation is lacking lead us to cobble our own procedure together in order to accomplish the goal of updating all repositories

What we needed to accomplish

• Manually create a new signing-service - the CLI nor API seem to provide this mechanic
• We then needed to update all repos to leverage the new signing key, the rpm plugin has a flag for this, so we can use the CLI, while the deb plugin does not, we needed to use the API.
• We finished this process off by creating a publication - luckily we can use the CLI for this, though I should call out that the deb and rpm modules have differing flags for this

The Pain Points

• There's no documentation about this. There's docs on creating and managing the key in the Operator, but there's no "here's how to update your key."
• The differences in how to perform the actions on repos created confusion
• The process is awkwardly sluggish - for the near 1900 repos we manage, this was a 5 hour process, admittedly our scripts could've been better, but we can iterate on this.
• The operator does not seem to be able to update the signing key - which I believe the Operator likely should not

Why does the Operator manage the signing key?

If we attempt to update the Operator with the new key, it doesn't actually work at all! Per the steps located in the Operator for this exact step, we edit the Pulp CR with our newly created secret objects. Neat. But what happens on the Pod that get's spun up fails.

...
CommandError: Signing service 'apt-signing-service' could not be removed because it's still in use.
CommandErrorCommandError: duplicate key value violates unique constraint "core_signingservice_name_key"
DETAIL:  Key (name)=(apt-signing-service) already exists.
CommandErrorCommandErrorCommandError: Signing service 'rpm-signing-service' of class 'rpm:RpmPackageSigningService' does not exists.
Signing service 'rpm-signing-service' has been successfully removed.
Successfully added signing service rpm-signing-service for key C2993C31C3D30FFB82E518EA63DEDDCC36BA97B0.

Notice a few things:

• We failed to do anything with the Apt key. Yes it is still used, we need to update it, so using the Operator here is not an option, thus the Release.gpg is still pointed to the old key.
• The rpm-signing-service wasn't actually removed, it still showed up when performing a signing-key list on the CLI
• Not shown in the logs above, for RPM based repos, you'd think we did the right thing here, but alas, if you go to any repo and grab the repomd.xml.key, it still represents the old key.

Shortfalls

Because the Operator is attempting to maintain this key, and it doesn't work, at least in our 2 environments, showcases a problem with general use of this key. The fact that we have a procedure that requires us to go back and update all existing repos further points that the Operator is likely not the right place for this key to be maintained. Because documentation doesn't exist creates some stress on Engineers maintaining this system. I'm not sure what the historical nature of this led to our current behavior, but minimally we certainly need some documentation around this.

So what did we do?

At the time of this writing, this was documented in the public over here: https://gitlab.com/gitlab-org/build/team-tasks/-/issues/157#note_3120509156

Start with the existing Operator documentation for creating and managing signing keys, then compare the CLI and API workflows described for the rpm and deb plugins. Done means documenting a reliable key-update procedure, including repository updates, publication, and the differences between rpm and deb operations.