knative / knative/serving

Bound Concurrency of Pod Scraping (Probably with a Work Pool)

Open
#8,377 29 comments 0 reactions 0 assignees View on GitHub
area/autoscale kind/feature lifecycle/frozen triage/accepted
Dominant language
Go
Stars
6.1k
Forks
1.2k
Avg merge
2d 7h
Merged PRs (30d)
2

Description

## Describe the feature

We've discussed this a few times in slack but it seems worth having an issue for it and I didn't spot one, so here one is :).

As of right now we spawn pod scrapes in parallel for ~every revision with no real bound on the amount we'll attempt at once across revisions. This means we can potentially have an extremely large number of active outgoing scrapes, causing more contention than needed, exceeding keep-alive connection pool size and buffer pools etc. In our larger environments, we see large numbers of connections waiting on DNS and GC when revisions*pods gets large.

We should probably introduce some sort of work pool for the scrapers.

Contributor guide

Open the contributing guide

Research direction

No file or test path is provided. Start by locating the pod-scraping entry point and tracing how scrapes are launched across revisions; determine where concurrency is currently unbounded. Done means concurrent outgoing scrapes are bounded across revisions, with behavior and limits covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.