openshift / openshift/origin

ProgressDeadlineExceeded cannot be extended

Open
#23,883 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

lifecycle/frozen
Dominant language
Go
Stars
8.7k
Forks
4.8k
Avg merge
4d 10h
Merged PRs (30d)
53

Description

Pods are getting killed after 100 minutes and marked as Failed progressing.
We have some big boys in our environment. Our image is about ~5GB and needs around 90-110 minutes to be up and running (1st deployment).
After approximately 100 mins our pod(s) is/are getting deleted without a reason, even though everything inside was going well.

Version

oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
openshift v3.11.69
kubernetes v1.11.0+d4cacc0

Steps To Reproduce
  1. Prepare DeploymentConfig with a ~5GB image
  2. Set all timeouts to 7200 (2hrs)
  3. Deploy it
Current Result

Example from one of the timeouts:

 conditions:
    - lastTransitionTime: '2019-09-27T05:33:28Z'
      lastUpdateTime: '2019-09-27T05:33:28Z'
      message: Deployment config does not have minimum availability.
      status: 'False'
      type: Available
    - lastTransitionTime: '2019-09-27T07:30:32Z'
      lastUpdateTime: '2019-09-27T07:30:32Z'
      message: replication controller "app-wls-1" has failed progressing
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing

We have been also trying to patch the DC and include it in YAML DC file, but it seems that Openshift is ignoring this spec in YAML (is it available only for deployments?). Patch command returns following output:

$ oc patch dc app --patch='{"spec":{"progressDeadlineSeconds":7200}}'
deploymentconfig.apps.openshift.io/app not patched
Expected Result

Successful deployment without exceeding any deadline. ;)

Additional Information
$ oc get all -o yaml -n szymon-sandbox >> namespace.yml

namespace.yml

$ oc describe rc/app-wls-1

rc.yml

Please, kindly advise. :) Feel free to ask me for any additional info or missing details.
Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked namespace.yml and rc.yml files alongside the DeploymentConfig patch command and the reported OpenShift and Kubernetes versions. Reproduce the deployment with the ~5GB image and 7200-second settings, then verify whether the DeploymentConfig accepts a configurable progress deadline and whether the pod remains running beyond 100 minutes.

Written by the indexing model from the issue text.

Assessment

Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.