linkedin / linkedin/parseq

Automated capacity management

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
268
PR merge metrics
No merged PRs in 30d

Description

Due to asynchronous nature Engine can accept for execution more plans than it is physically capable of executing. A partial answer to this problem is to use specify plan concurrency and apply back pressure to incoming plans. The problems with this approach are:
- inefficiency in situations where different plan types have very different resource utilizations
- difficult to figure out correct concurrency level
- best concurrency level may change over time

This task is to investigate automatic capacity management. Some ideas:
- in real time, measure utilization of key resources and and distribute cost over plan types being executed
- estimate load based on sample of traces of executed plans and correlate it with resource utilization

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by reviewing the existing plan-concurrency and back-pressure behavior, then investigate how execution traces and key-resource utilization could be measured and related to plan types. Done means producing a concrete automatic capacity-management approach that addresses changing workloads and differing resource use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.