pulp / pulp/pulp-operator

[BUG] Upload/download content failures during pod rollouts

Open
#978 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Issue
Dominant language
Go
Stars
88
Forks
66
Avg merge
53m
Merged PRs (30d)
1

Description

Describe the bug
Our pods are not getting terminated in a graceful way. An image version upgrade or any change that requires deploying new pods will cause a rollout of the Deployments and the pods will be terminated even in the middle of a running task (download/upload content).

To Reproduce
Steps to reproduce the behavior:
Install the operator with 2 api and content replicas:

spec:
  api:
    replicas: 2
  content:
    replicas: 2
  image_version: latest
  image_web_version: latest

start to upload a file:

pulp file content upload --file Fedora-Workstation-Live-x86_64-35-1.2.iso --relative-path Fedora-Workstation-Live-x86_64-35-1.2.iso

while the upload is running, modify Pulp CR to "force" the redeploy of pods:

spec:
  image_version: stable
  image_web_version: stable

pulp client will receive the following error:

.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 200
.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
uploads_delete : delete https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 204
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Expected behavior
The pods should not be terminated if there is/are any task(s) running.

Additional context
Verify which signal kubelet sends to the process in these scenarios.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Pulp CR deployment rollout described in the reproduction steps and investigate the kubelet termination signal during an active upload or download. Reproduce the rollout with two API and content replicas, then verify that active tasks complete without a client connection failure before considering the issue done.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.