Timeout on probe using container command doesn't work. Will CRI-O fix it.
@runcom is already working on this.
Since Feb 27, 2018.
- Dominant language
- Go
- Stars
- 8.7k
- Forks
- 4.8k
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 53
Description
I have been told previously long time back that this issue cannot be fixed so long as docker is used and so it may not have had an issue created for it. Am creating an issue for it now as want to know whether CRI-O will fix it.
The problem is that when you use a container command as a readiness and liveness probe, the timeout value for the probe doesn't work. That is, if the probe hangs, or takes a long time to run, then it will not be failed when the timeout expires, nor will the command for the probe be killed.
In the case of the probe taking a long time to run, once it does return, normal period between probes will then occur and probe will be run again. If however the probe hangs and never returns, it is never marked as failed, nor will subsequent probes ever run. So although the probe is failing, the pod will never be marked as failed and be restarted.
I have been told this problem can't be fixed because docker exec doesn't support a timeout value so that a command can be interrupted.
Is this going to be fixed by CRI-O, or are timeouts on probes when using a container command never going to be supported.
RIght now been advising people to implement their own timeout on probe execution in their probe script, or simply avoid container commands for probe scripts.
Version
oc v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
features: Basic-Auth
Server https://api.pro-us-east-1.openshift.com:443
openshift v3.6.173.0.21
kubernetes v1.6.1+5115d708d7
Steps To Reproduce
Create a liveness probe which uses a container command and have the command run sleep for a long period.
For example:
oc set probe dc/blog --liveness -- sleep 300
Current Result
Nothing happens after the default one second timeout on the probe. No event indicating failure of probe. You can get into the container and see the first probe is running:
$ ps aux | grep sleep
1004820+ 49 0.0 0.0 5888 612 ? Ss 01:18 0:00 sleep 300
1004820+ 99 0.0 0.0 10648 968 ? S+ 01:21 0:00 grep sleep
The process ID for the sleep never changes so is same process and not being killed nor subsequent probe run, at least not until the sleep finishes.
Expected Result
Should register a failure after one second. After two subsequent failures, the pod should be restarted.
Additional Information
None.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.