pulp / pulp/pulp-operator

Workers stuck due to nonexistent migrations

Open
#1,228 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Go
Stars
88
Forks
66
Avg merge
53m
Merged PRs (30d)
1

Description

Version
Pulp Operator: 1.0.0-beta.3
(The following are both important to note, as I'll describe later.)
Affected node: pulp-minimal:stable@0f66fa60566b
Unaffected node: pulp-minimal:stable@4b30cd0edbb7

Describe the bug
I'm not sure if this belongs in the operator, oci-images, or another repo, so let me know if I should open it somewhere else.
We deployed a Pulp in our K8s cluster and everything was fine. A few days later, I went to bump the worker replicas, but the new pods just sat with the following in their logs:

error: Failed to initialize NSS library
Database migration in progress. Waiting...

After a while (and stumbling across this issue), it occurred to me that the new pods were probably on nodes that hadn't previously pulled the minimal image, and sure enough the digest IDs for the images on both nodes didn't match. The API pod was on the "old" image... the new pods were on the "new" one. So, unless I'm misunderstanding what's happening here, these workers will never come up until the API pod is refreshed (with imagePullPolicy either set to Always, or with "unaffected" nodes cordoned, to ensure it'll come up on the newer image).

To Reproduce
Steps to reproduce the behavior:

  1. Deploy Pulp operator and a Pulp instance.
  2. Wait a few days (or until there's a new re-tag of pulp-minimal:latest).
  3. Cordon nodes currently running workers.
  4. Scale workers up and ensure they come up on "fresh" nodes with the newer image.

Expected behavior
I'm not sure... it's easy enough for the user to be more explicit with the images they want to use. On the other hand, that involves providing a bunch of values in the Pulp spec that can easily be mixed up. It seems like the Operator should recognize that a "new" image exists and update all the pods (in rolling fashion, of course), to ensure the migrations actually happen.

Additional context
None... but I wanted to say thanks for all the work that goes into Pulp!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Pulp Operator's handling of API and worker pod images, starting with the deployment and worker-scaling reproduction steps in the issue. Compare image digests and migration-waiting logs when new workers use a newer pulp-minimal image than the API pod. Done means newly created workers do not remain blocked behind inconsistent image migrations, with the behavior verified through an appropriate operator test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.