Workers stuck due to nonexistent migrations
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 88
- Forks
- 66
- Avg merge
- 53m
- Merged PRs (30d)
- 1
Description
Version
Pulp Operator: 1.0.0-beta.3
(The following are both important to note, as I'll describe later.)
Affected node: pulp-minimal:stable@0f66fa60566b
Unaffected node: pulp-minimal:stable@4b30cd0edbb7
Describe the bug
I'm not sure if this belongs in the operator, oci-images, or another repo, so let me know if I should open it somewhere else.
We deployed a Pulp in our K8s cluster and everything was fine. A few days later, I went to bump the worker replicas, but the new pods just sat with the following in their logs:
error: Failed to initialize NSS library
Database migration in progress. Waiting...
After a while (and stumbling across this issue), it occurred to me that the new pods were probably on nodes that hadn't previously pulled the minimal image, and sure enough the digest IDs for the images on both nodes didn't match. The API pod was on the "old" image... the new pods were on the "new" one. So, unless I'm misunderstanding what's happening here, these workers will never come up until the API pod is refreshed (with imagePullPolicy either set to Always, or with "unaffected" nodes cordoned, to ensure it'll come up on the newer image).
To Reproduce
Steps to reproduce the behavior:
- Deploy Pulp operator and a Pulp instance.
- Wait a few days (or until there's a new re-tag of pulp-minimal:latest).
- Cordon nodes currently running workers.
- Scale workers up and ensure they come up on "fresh" nodes with the newer image.
Expected behavior
I'm not sure... it's easy enough for the user to be more explicit with the images they want to use. On the other hand, that involves providing a bunch of values in the Pulp spec that can easily be mixed up. It seems like the Operator should recognize that a "new" image exists and update all the pods (in rolling fashion, of course), to ensure the migrations actually happen.
Additional context
None... but I wanted to say thanks for all the work that goes into Pulp!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the Pulp Operator's handling of API and worker pod images, starting with the deployment and worker-scaling reproduction steps in the issue. Compare image digests and migration-waiting logs when new workers use a newer pulp-minimal image than the API pod. Done means newly created workers do not remain blocked behind inconsistent image migrations, with the behavior verified through an appropriate operator test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100