kubermatic / kubermatic/dashboard

Test dashboard API metrics endpoint error handling of BYO CNI clusters - Test Release 2.31

Open
#8,063 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/rotten sig/ui
Dominant language
Go
Stars
50
Forks
73
Avg merge
2d 22h
Merged PRs (30d)
31

Description

### Summary
There was a bug within the Dashboard API where the user cluster metrics endpoints returned a 503 response and an corresponding error notification was shown within the Dashboard UI periodically (every ~10 seconds) when browsing the details view of a user cluster that was created with CNI set to `none` (BYO CNI mode) but no CNI was set up for the cluster manually.
As a consequence the API error rate limit increased, eventually triggering the [KubermaticAPITooManyErrors](https://github.com/mgoltzsche/kubermatic/blob/d969847a31361de981195d745e0fd44c8355486b/charts/monitoring/prometheus/rules/kubermatic-master-kubermatic.yaml#L29) alert.
(The description of the [PR](https://github.com/kubermatic/dashboard/pull/8060) provides screenshots illustrating all of that.)

The unavailability is of the metrics endpoints is actually expected in that case as long as the user didn't set up a CNI provider herself.
Therefore the fix was to let the metrics endpoint return a 200 status with empty metrics in that case.

The goal of this ticket is to verify that the fix works as expected.

**Testing scope:**

**Related:**
- Issue: https://github.com/kubermatic/kubermatic/issues/15801
- PR: https://github.com/kubermatic/dashboard/pull/8060

### Type of Testing
- [ ] New Feature
- [x] Bug Fix / Regression
- [ ] UI/UX
- [ ] Performance
- [ ] Security / Permissions (RBAC)
- [ ] Upgrade / Migration
- [ ] Other:

### Prerequisites

**Environment:**
- [ ] Provider Specific Setup (If Applicable):

### Test Scenarios

- [ ] **Scenario 1: Unavailable BYO CNI user cluster**
- **Steps:**
1. Create a user cluster with BYO CNI (select `none` as CNI provider).
2. Don't install the CNI and let the user cluster worker nodes be in a "NotReady" state.
3. Browse the detail view of that user cluster within the KKP Dashboard UI.
- **Expected:**
- The "the server is currently unable to handle the request" notification should not show up within the KKP Dashboard UI anymore when browsing the detail view of that user cluster.
- The KKP Dashboard API server logs the error as a warning (not error) periodically: "the server is currently unable to handle the request".
- The kubermatic-api request error rate should not increase, even if more of those BYO CNI clusters are created and their dashboard views opened in several browser tabs in parallel. The error rate (`sum(rate(http_requests_total{app_kubernetes_io_name="kubermatic-api",code=~"5.."}[5m]))`) can be observed within prometheus (for comparison, see [screenshot within PR description](https://github.com/kubermatic/dashboard/pull/8060)).

### Edge Cases & Boundary Conditions

- [ ]

### Screenshots / Attachments

### Acceptance Criteria

- [ ]

### Notes

### Test Environment
- **UI Version:**
- **API Version:**
- **K8s Version:**
- **Provider:**
- **Browser & OS:**
- **Domain:**

Contributor guide

Open the contributing guide

Research direction

Start with the linked dashboard PR 8060 and related issue 15801, then create a user cluster with CNI set to none and no manually installed CNI. Open its detail view and monitor the Dashboard UI, API logs, and the Prometheus http_requests_total query. Done means no UI error notification appears, the periodic message is logged as a warning, and the kubermatic-api 5xx error rate does not increase.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
backend, observability, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.