knative / knative/serving

Activator unnecessarily probing when rolling out a new Revision with long startup times

Open
#13,611 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/question triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

Ask your question here:

Hi there,
When a service starts up, in our case we need a few seconds for the application to be ready. There are two challenges we are facing:
1.

{"severity":"INFO","timestamp":"2023-01-16T14:33:40.72789972Z","logger":"queueproxy","caller":"sharedmain/main.go:270","message":"Starting queue-proxy","commit":"d7e9873","knative.dev/key":"bc878a0622175693055adcfae0fd8cdf22cfdsfs10","knative.dev/pod":"bc878a0622175693055adcfae0fd8cdf22cfdsfs10-deployment-g2q5f"}
aggressive probe error (failed 202 times): Get "http://127.0.0.1:8080/healthz": dial tcp 127.0.0.1:8080: connect: connection refused
timed out waiting for the condition
{"severity":"INFO","timestamp":"2023-01-16T15:58:39.370386227Z","logger":"queueproxy","caller":"sharedmain/main.go:270","message":"Starting queue-proxy","commit":"d7e9873","knative.dev/key":"3d495013","knative.dev/pod":"3d495013-deployment-7d984f94c8-jw2sw"}
Get "http://127.0.0.1:8080/healthz": dial tcp 127.0.0.1:8080: connect: connection refused
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500
HTTP probe did not respond Ready, got status code: 500

Is there any way to setup queue-proxy readinessProbes to be a bit more patient? I mean something like the initialDelaySeconds with the services.

Thanks in advance!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The logs identify sharedmain/main.go:270 and the queue-proxy/Activator readiness-probe path; start by tracing how readinessProbes are configured and how long-starting containers are handled. Reproduce a Revision with delayed startup, then verify that probing remains patient enough for the application to become ready without premature failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.