kubermatic / kubermatic/machine-controller

Reworking E2E tests for Machine Controller to support external CCM

Open
#1,626 7 comments 1 reaction 0 assignees View on GitHub
kind/chore kind/failing-test lifecycle/frozen priority/high
Dominant language
Go
Stars
326
Forks
138
Avg merge
2d 46m
Merged PRs (30d)
12

Description

Kubernetes maintainers have long been discouraging the usage of In-tree cloud controller manager(CCM) in favor of external CCM. In-tree providers are planned to be removed from the Kubernetes core and we are already affected with:
- OpenStack on k8s v1.26+
- AWS on k8s v1.27+

For now, we are ignoring or skipping e2e tests for the affected versions but this is not ideal/acceptable, it's important to ensure that the e2e tests are reliable and up-to-date.

## Proposed solution

**Switch from KIND to KKP for machine-controller E2E tests.**

1. Create a dedicated KKP setup(or dedicated seed) for machine-controller. Connectivity for nodes would work out of the box and we won't need any hacks for CCM or CNI.
2. Creating user clusters for e2e tests. We have multiple options for that; we should evaluate and choose:
1. Long-standing user clusters per cloud provider with no machine deployments. MC will create the required machine deployments for tests and then clean them up.
2. When running tests, create a user cluster per cloud provider. We can check if a user cluster for the provider already exists and then just use it.
3. Dedicated user cluster per CI job; this is not ideal due to resource constraints and the cost incurred.
3. Run our test suite to check that the machine came up and was ready. We don't need conformance tests etc. here.

This would help solve https://github.com/kubermatic/machine-controller/issues/1462 as well.

**NOTE: Proper cleanup jobs should be in place right from the start to avoid broken/cancelled jobs leading to orphaned resources**

## Alternate solution

- Stick to KIND and deploy external CCM for each cloud provider when it's required
- Create single node clusters using KubeOne to run the tests

Contributor guide

Open the contributing guide

Research direction

Start by evaluating the proposed switch from KIND to KKP and the three user-cluster strategies described in the issue. Check how the machine-controller E2E suite currently runs, then define the required external CCM setup and cleanup jobs. Done means the suite reliably verifies machine readiness across affected cloud providers without orphaned resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ci-cd, infrastructure, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.