stackabletech / stackabletech/stackablectl

GVK resolution fails if metrics-server is unavailable

Open
#335 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Rust
Stars
10
Forks
5
Avg merge
4h 41m
Merged PRs (30d)
4

Description

Affected version

stackablectl 24.7.1

Current and expected behavior
  1. Run kubectl -n kube-system delete pods -l k8s-app=metrics-server && stackablectl release install dev
  2. Observe that stackablectl crashes with the following error message:
  ERROR  failed with status 503 Service Unavailable
    at src/client/builder.rs:199

   WARN  Unsuccessful data error parse: service unavailable

    at src/client/mod.rs:467

An unrecoverable error occured: failed to execute release (sub)command

Caused by these errors (recent errors listed first):
 1: failed to create Kubernetes client
 2: failed to run GVK discovery
 3: ApiError: "service unavailable\n": Failed to parse error data (ErrorResponse { status: "503 Service Unavailable", message: "\"service unavailable\\n\"", reason: "Failed to parse error data", code: 503 })
 4: "service unavailable\n": Failed to parse error data
  1. Observe that kubectl and k9s are able to manage the cluster just fine

I suspect that this comes down to metrics-server using K8s API aggregation, which allows it to provide a fake "resource" that is stored by itself rather than in etcd. This also means that that resource can be unavailable even if the apiserver and etcd are both doing fine.

Possible solution

We could either:

  1. Limit GVK resolution to the apigroups we care about
  2. Defer apigroup-specific resolution errors until accessing the relevant apigroup
Additional context

No response

Environment

Using k3s v1.31.0+k3s1 via k3d

Would you like to work on fixing this bug?

None

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with kubectl -n kube-system delete pods -l k8s-app=metrics-server && stackablectl release install dev, then inspect GVK discovery from src/client/builder.rs:199 and error handling around src/client/mod.rs:467. The fix should let release installation continue when metrics-server is unavailable while preserving resolution for the API groups stackablectl needs.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
api, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.