knative / knative/test-infra

wait_until_pods_running should be able to rule out error pods that are not related to the final state of the owners

Open
#1,611 8 comments 0 reactions 0 assignees View on GitHub
bug lifecycle/frozen
Dominant language
Go
Stars
83
Forks
160
PR merge metrics
No merged PRs in 30d

Description

In https://github.com/knative/test-infra/blob/97901db33f62f8e1abe9c8221670a09c5a1cba8a/scripts/library.sh#L133, `wait_until_pods_running` will only succeed if all pods in the given namespace are in `Running` or `Completed` state.

But since k8s has some retry logic, e.g. K8s Job can create a new pod if there is an error, one error pod does not necessarily mean the Job fails - https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/6440/pull-knative-serving-integration-tests/1214219978266382337 is an example. In such scenario `wait_until_pods_running` will return an error that is not necessarily true.

This function should be general enough to consider and rule out error pods that are not related to the final state of the owners, e.g.
1. For Deployments, all pods should be X/X Running
2. For Jobs, it should depend on the success criteria
...

FYI @mattmoor

Contributor guide

Open the contributing guide

Research direction

Start in scripts/library.sh around line 133 and trace wait_until_pods_running's pod-state checks. Review the linked Kubernetes Job retry example and determine how owner types and their final-state criteria should be represented. Done means the function can distinguish transient error pods from failures while preserving the stated Deployment and Job behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, shell
Domain
infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.