Queing requests with 2 ready pods and containerConcurrency=1, number of activators = 2
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 1.2k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 2
Description
## What version of Knative?
1.6.0
## Expected Behavior
On a service with `target-burst-capacity: -1` (activator always in path), and `containerConcurrency: 1`, I would expect that if I have a ready pod and a new request comes in, said ready pod would handle that request. In particular, if pod A is already handling a request (call it `r1`) and pod B is idle, I would expect an incoming request `r2` to be handled by pod B without delay.
## Actual Behavior
`r2` is not handled by pod B when it arrives, but is queued until pod A finishes handling `r1` **when there are multiple activators**
## Steps to Reproduce the Problem
1. Install Knative Serving with any networking layer (issue was reproduced with kourier and istio)
1. Scale the data plane so there are two activators: `kubectl patch hpa activator -n knative-serving -p '{"spec":{"minReplicas":2,"maxReplicas":19}}'`
1. Deploy a service with tbc=-1, cc=1 that takes some time to process each request. For example: https://github.com/psschwei/delayed/blob/master/service.yaml and can be deployed with `ko apply -f service.yaml` (note this app will receive a request, sleep for 30 seconds by default, and then return successfully)
1. Send a request to the service: `curl $(kn service describe delayed -ourl) &`
1. Wait for a second pod to become ready
1. Send another request to the service: `curl $(kn service describe delayed -ourl) &`
1. Observe that the second request is not processed by the waiting pod
Aggregate view of the logs while this is happening (`.13` is kourier, `.6` and `.18` are my activators):
```bash
$ stern delayed -c user-container &
+ delayed-00001-deployment-54b7cb598f-m4q7b › user-container
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 starting server
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 server ready
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 listening on port 8080
# 1 pod is ready
# now let's send a request
$ curl $(kn service describe delayed -ourl) &
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:43 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:43 inbound path: [10.244.0.13, 10.244.0.6]
+ delayed-00001-deployment-54b7cb598f-phsn5 › user-container
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 starting server
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 server ready
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 listening on port 8080
# second pod is created and ready
# send a new request, happens between 20:37:44 and 20:37:52
$ curl $(kn service describe delayed -ourl) &
+ delayed-00001-deployment-54b7cb598f-jcpv8 › user-container
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 starting server
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 server ready
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 listening on port 8080
# 3rd pod scales up, but we don't see the second request being handled
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 Request handled
# first request finishes, now the second request is handled by a different pod
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 inbound path: [10.244.0.13, 10.244.0.18]
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:44 Request handled
# second request done
- delayed-00001-deployment-54b7cb598f-jcpv8 › user-container
- delayed-00001-deployment-54b7cb598f-phsn5 › user-container
# scale down
# let's try it again
$ curl $(kn service describe delayed -ourl) &
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:00 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:00 inbound path: [10.244.0.13, 10.244.0.6]
delayed-00001-deployment-54b7cb598f-sz6bh › user-container
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 starting server
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 server ready
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 listening on port 8080
# second pod ready again, let's send request 2 (betweeen 20:44:01 and 20:44:16)
$ curl $(kn service describe delayed -ourl) &
+ delayed-00001-deployment-54b7cb598f-sxz8d › user-container
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 starting server
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 server ready
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 listening on port 8080
# 3rd pod scaled up, but 2nd request is stuck in the queue
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:30 Request handled
# 1st request finishes, now the second one is handled, this time by the same pod
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:30 Starting to handle request
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:30 inbound path: [10.244.0.13, 10.244.0.6]
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:45:00 Request handled
- delayed-00001-deployment-54b7cb598f-sxz8d › user-container
- delayed-00001-deployment-54b7cb598f-sz6bh › user-container
# request done and scale down
```
Contributor guide
Research direction
Reproduce on Knative Serving 1.6 with two activators, target-burst-capacity -1, containerConcurrency 1, and the delayed service from the linked manifest. Start by tracing the activator request path and the aggregate logs described in the issue; done means a second request is handled by an idle ready pod instead of waiting for the first request to finish.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100