Improve parallel catchup v2 resource consumption

Open
#397 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
fsharp, kubernetes
Domain
distributed-systems, infrastructure

Research direction

Start by tracing the parallel catchup v2 mission and its worker-pod lifecycle. Review how pods are created, jobs retried, and pods torn down; done means idle workers can be released before mission completion and stuck jobs can be retried earlier without leaving unnecessary Kubernetes capacity running.

Written by the indexing model from the issue text.

Description

What problem does your feature solve?

Currently the parallel catchup v2 mission works in this way:

  1. Spin up n worker pods
  2. Workers pick up jobs
  3. When all jobs are complete, or when the mission fails, the worker pods are torn down

This can cause excessive resource consumption. For example

  1. We spin up 1024 worker pods. This results in dynamic provisioning of k8s workers (possibly hundreds of them depending on instance type)
  2. If a job gets stuck it will be retried, but only once all other jobs are finished
  3. The stuck job may take a while to complete. During this time we keep all worker pods, and k8s workers, online, even though only one job is running

Another scenario is where a significant portion of the jobs finish. We may have 50% of the worker pods doing some work, yet we keep all of them up.

What would you like to see?

It would be good for the worker pods to be torn down even before the mission finishes. Once they are torn down karpenter should be able to release some workers to reduce cost.
Perhaps we can also improve retry logic to catch dead jobs earlier. If we can retry earlier in the run we'll avoid long tail.

What alternatives are there?
Dominant language
F#
Stars
10
Forks
22
Avg merge
1d 1h
Merged PRs (30d)
3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stellar/supercluster

All issues in stellar/supercluster

Similar issues

More Distributed Systems issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.