Activator unnecessarily probing when rolling out a new Revision with long startup times
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
Ask your question here:
Hi there,
When a service starts up, in our case we need a few seconds for the application to be ready. There are two challenges we are facing:
1.
{"severity":"INFO","timestamp":"2023-01-16T14:33:40.72789972Z","logger":"queueproxy","caller":"sharedmain/main.go:270","message":"Starting queue-proxy","commit":"d7e9873","knative.dev/key":"bc878a0622175693055adcfae0fd8cdf22cfdsfs10","knative.dev/pod":"bc878a0622175693055adcfae0fd8cdf22cfdsfs10-deployment-g2q5f"}
aggressive probe error (failed 202 times): Get "http://127.0.0.1:8080/healthz": dial tcp 127.0.0.1:8080: connect: connection refused
timed out waiting for the condition
{"severity":"INFO","timestamp":"2023-01-16T15:58:39.370386227Z","logger":"queueproxy","caller":"sharedmain/main.go:270","message":"Starting queue-proxy","commit":"d7e9873","knative.dev/key":"3d495013","knative.dev/pod":"3d495013-deployment-7d984f94c8-jw2sw"}
Get "http://127.0.0.1:8080/healthz": dial tcp 127.0.0.1:8080: connect: connection refused
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
Is there any way to setup queue-proxy readinessProbes to be a bit more patient? I mean something like the initialDelaySeconds with the services.
Thanks in advance!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The logs identify sharedmain/main.go:270 and the queue-proxy/Activator readiness-probe path; start by tracing how readinessProbes are configured and how long-starting containers are handled. Reproduce a Revision with delayed startup, then verify that probing remains patient enough for the application to become ready without premature failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100