openshift / openshift/origin

Deployment can gets stuck after recreating

Open
#14,542 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/apps kind/bug lifecycle/frozen priority/P2
Dominant language
Go
Stars
8.7k
Forks
4.8k
Avg merge
4d 10h
Merged PRs (30d)
53

Description

After running this little script the output will be following:

--> running test #1 ...
deploymentconfig "test" deleted
deploymentconfig "test" created
--> status of #1 is Pending ...
--> status of #1 is Running ...
--> running test #2 ...
deploymentconfig "test" deleted
pod "test-1-deploy" deleted
pod "test-1-s7vq4" deleted
deploymentconfig "test" created
--> status of #2 is Running ...
--> running test #3 ...
deploymentconfig "test" deleted
pod "test-1-deploy" deleted
deploymentconfig "test" created
--> status of #3 is Failed ...

What I think happen is that the "test-1-deploy" pod is in Terminating status (or it is being terminated by GC), however it is not fast enough and that means the next DC will fail to create deployer pod (because it already exists) and it never retries.

Before we giving up on creating a new deployer pod for new DC, we should perhaps check the ownerRef and verify that the test-1-deploy which already exists is meant to be for the current DC. We should reconcile and recreate that Pod for the new DC?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the deployment recreation sequence with the linked script and observe the deployer pod lifecycle. Trace the deployment controller's deployer-pod creation and reconciliation path, then verify owner references and retry behavior when the previous pod is still terminating or already exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.