wait_until_pods_running should be able to rule out error pods that are not related to the final state of the owners
- Dominant language
- Go
- Stars
- 83
- Forks
- 160
- PR merge metrics
- No merged PRs in 30d
Description
In https://github.com/knative/test-infra/blob/97901db33f62f8e1abe9c8221670a09c5a1cba8a/scripts/library.sh#L133, `wait_until_pods_running` will only succeed if all pods in the given namespace are in `Running` or `Completed` state.
But since k8s has some retry logic, e.g. K8s Job can create a new pod if there is an error, one error pod does not necessarily mean the Job fails - https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/6440/pull-knative-serving-integration-tests/1214219978266382337 is an example. In such scenario `wait_until_pods_running` will return an error that is not necessarily true.
This function should be general enough to consider and rule out error pods that are not related to the final state of the owners, e.g.
1. For Deployments, all pods should be X/X Running
2. For Jobs, it should depend on the success criteria
...
FYI @mattmoor
Contributor guide
Research direction
Start in scripts/library.sh around line 133 and trace wait_until_pods_running's pod-state checks. Review the linked Kubernetes Job retry example and determine how owner types and their final-state criteria should be represented. Done means the function can distinguish transient error pods from failures while preserving the stated Deployment and Job behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, shell
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100