[BUG] Upload/download content failures during pod rollouts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 88
- Forks
- 66
- Avg merge
- 53m
- Merged PRs (30d)
- 1
Description
Describe the bug
Our pods are not getting terminated in a graceful way. An image version upgrade or any change that requires deploying new pods will cause a rollout of the Deployments and the pods will be terminated even in the middle of a running task (download/upload content).
To Reproduce
Steps to reproduce the behavior:
Install the operator with 2 api and content replicas:
spec:
api:
replicas: 2
content:
replicas: 2
image_version: latest
image_web_version: latest
start to upload a file:
pulp file content upload --file Fedora-Workstation-Live-x86_64-35-1.2.iso --relative-path Fedora-Workstation-Live-x86_64-35-1.2.iso
while the upload is running, modify Pulp CR to "force" the redeploy of pods:
spec:
image_version: stable
image_web_version: stable
pulp client will receive the following error:
.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 200
.uploads_update : put https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
uploads_delete : delete https://example-pulp.apps-crc.testing/pulp/api/v3/uploads/0188c54f-6ab8-7c75-9f6c-5014d0e0670b/
Response: 204
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Expected behavior
The pods should not be terminated if there is/are any task(s) running.
Additional context
Verify which signal kubelet sends to the process in these scenarios.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Pulp CR deployment rollout described in the reproduction steps and investigate the kubelet termination signal during an active upload or download. Reproduce the rollout with two API and content replicas, then verify that active tasks complete without a client connection failure before considering the issue done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100