knative / knative/serving

Queing requests with 2 ready pods and containerConcurrency=1, number of activators = 2

Open
#13,331 4 comments 1 reaction 0 assignees View on GitHub
kind/bug triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

## What version of Knative?

1.6.0

## Expected Behavior

On a service with `target-burst-capacity: -1` (activator always in path), and `containerConcurrency: 1`, I would expect that if I have a ready pod and a new request comes in, said ready pod would handle that request. In particular, if pod A is already handling a request (call it `r1`) and pod B is idle, I would expect an incoming request `r2` to be handled by pod B without delay.

## Actual Behavior

`r2` is not handled by pod B when it arrives, but is queued until pod A finishes handling `r1` **when there are multiple activators**

## Steps to Reproduce the Problem

1. Install Knative Serving with any networking layer (issue was reproduced with kourier and istio)

1. Scale the data plane so there are two activators: `kubectl patch hpa activator -n knative-serving -p '{"spec":{"minReplicas":2,"maxReplicas":19}}'`

1. Deploy a service with tbc=-1, cc=1 that takes some time to process each request. For example: https://github.com/psschwei/delayed/blob/master/service.yaml and can be deployed with `ko apply -f service.yaml` (note this app will receive a request, sleep for 30 seconds by default, and then return successfully)

1. Send a request to the service: `curl $(kn service describe delayed -ourl) &`

1. Wait for a second pod to become ready

1. Send another request to the service: `curl $(kn service describe delayed -ourl) &`

1. Observe that the second request is not processed by the waiting pod

Aggregate view of the logs while this is happening (`.13` is kourier, `.6` and `.18` are my activators):

```bash
$ stern delayed -c user-container &

+ delayed-00001-deployment-54b7cb598f-m4q7b › user-container
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 starting server
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 server ready
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:02 listening on port 8080

# 1 pod is ready
# now let's send a request
$ curl $(kn service describe delayed -ourl) &

delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:43 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:37:43 inbound path: [10.244.0.13, 10.244.0.6]
+ delayed-00001-deployment-54b7cb598f-phsn5 › user-container
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 starting server
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 server ready
delayed-00001-deployment-54b7cb598f-phsn5 user-container 2022/09/21 20:37:44 listening on port 8080

# second pod is created and ready
# send a new request, happens between 20:37:44 and 20:37:52
$ curl $(kn service describe delayed -ourl) &

+ delayed-00001-deployment-54b7cb598f-jcpv8 › user-container
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 starting server
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 server ready
delayed-00001-deployment-54b7cb598f-jcpv8 user-container 2022/09/21 20:37:52 listening on port 8080

# 3rd pod scales up, but we don't see the second request being handled

delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 Request handled

# first request finishes, now the second request is handled by a different pod

delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:14 inbound path: [10.244.0.13, 10.244.0.18]
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:38:44 Request handled

# second request done

- delayed-00001-deployment-54b7cb598f-jcpv8 › user-container
- delayed-00001-deployment-54b7cb598f-phsn5 › user-container

# scale down

# let's try it again

$ curl $(kn service describe delayed -ourl) &

delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:00 Starting to handle request
delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:00 inbound path: [10.244.0.13, 10.244.0.6]
delayed-00001-deployment-54b7cb598f-sz6bh › user-container
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 starting server
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 server ready
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:01 listening on port 8080

# second pod ready again, let's send request 2 (betweeen 20:44:01 and 20:44:16)

$ curl $(kn service describe delayed -ourl) &

+ delayed-00001-deployment-54b7cb598f-sxz8d › user-container
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 starting server
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 server ready
delayed-00001-deployment-54b7cb598f-sxz8d user-container 2022/09/21 20:44:16 listening on port 8080

# 3rd pod scaled up, but 2nd request is stuck in the queue

delayed-00001-deployment-54b7cb598f-m4q7b user-container 2022/09/21 20:44:30 Request handled

# 1st request finishes, now the second one is handled, this time by the same pod

delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:30 Starting to handle request
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:44:30 inbound path: [10.244.0.13, 10.244.0.6]
delayed-00001-deployment-54b7cb598f-sz6bh user-container 2022/09/21 20:45:00 Request handled
- delayed-00001-deployment-54b7cb598f-sxz8d › user-container
- delayed-00001-deployment-54b7cb598f-sz6bh › user-container

# request done and scale down

```

Contributor guide

Open the contributing guide

Research direction

Reproduce on Knative Serving 1.6 with two activators, target-burst-capacity -1, containerConcurrency 1, and the delayed service from the linked manifest. Start by tracing the activator request path and the aggregate logs described in the issue; done means a second request is handled by an idle ready pod instead of waiting for the first request to finish.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.