stackabletech / stackabletech/stackablectl

Throttle requests when api-server tries to rate limit stackablectl

Open
#370 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
10
Forks
5
Avg merge
4h 41m
Merged PRs (30d)
4

Description

When trying to run stacklet list an an AWS eks cluster recently (replicated cluster, k8s 1.32) - the api-server was apparently unhappy about the rate at which requests were sent to retrieve details for Stackable objects.

soenke-desktop:~$ stackablectl stacklet list
An unrecoverable error occured: failed to execute stacklet (sub)command

Caused by these errors (recent errors listed first):
 1: failed to list stacklets
 2: failed to fetch data from the Kubernetes API
 3: failed to fetch data from Kubernetes API
 4: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 })
 5: storage is (re)initializing: TooManyRequests

We should probably somehow honour 429 status codes and throttle requests in that case. Or maybe implement this upstream in kube-rs.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the request path used by the stackablectl stacklet list command and trace how Kubernetes API errors are handled. Check whether kube-rs already exposes retry or throttling behavior for HTTP 429 responses, and reproduce the failure against an EKS cluster if possible. Done means the command handles the reported rate-limit condition without failing immediately, with the chosen behavior verified by tests or reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
api, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.