kubermatic / kubermatic/machine-controller

vSphere cloudProvider - failed to get folder

Open
#1,820 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug kind/feature priority/normal sig/cluster-management
Dominant language
Go
Stars
326
Forks
138
Avg merge
2d 46m
Merged PRs (30d)
12

Description

What happened?

I'm trying to setup a k8s cluster via KubeOne with the vSphere cloudProvider.
The setup of the VMs is done via Terraform, see the output of the command terraform output -json > tf.json below:

# tf.json
{
  "kubeone_api": {
    "sensitive": false,
    "type": [
      "object",
      {
        "apiserver_alternative_names": [
          "list",
          "string"
        ],
        "endpoint": "string"
      }
    ],
    "value": {
      "apiserver_alternative_names": [],
      "endpoint": "x.x.x.x"
    }
  },
  "kubeone_hosts": {
    "sensitive": false,
    "type": [
      "object",
      {
        "control_plane": [
          "object",
          {
            "bastion": "string",
            "bastion_host_key": "string",
            "bastion_port": "number",
            "bastion_user": "string",
            "cloud_provider": "string",
            "cluster_name": "string",
            "hostnames": [
              "list",
              "string"
            ],
            "private_address": [
              "tuple",
              []
            ],
            "public_address": [
              "tuple",
              [
                "string",
                "string",
                "string"
              ]
            ],
            "ssh_agent_socket": "string",
            "ssh_hosts_keys": [
              "list",
              "string"
            ],
            "ssh_port": "number",
            "ssh_private_key_file": "string",
            "ssh_user": "string"
          }
        ]
      }
    ],
    "value": {
      "control_plane": {
        "bastion": "",
        "bastion_host_key": null,
        "bastion_port": 22,
        "bastion_user": "",
        "cloud_provider": "vsphere",
        "cluster_name": "kkp-test",
        "hostnames": [
          "kkp-test-cp-1",
          "kkp-test-cp-2",
          "kkp-test-cp-3"
        ],
        "private_address": [],
        "public_address": [
          "x.x.x.x",
          "x.x.x.x",
          "x.x.x.x"
        ],
        "ssh_agent_socket": "env:SSH_AUTH_SOCK",
        "ssh_hosts_keys": null,
        "ssh_port": 22,
        "ssh_private_key_file": "",
        "ssh_user": "kubeone"
      }
    }
  },
  "kubeone_workers": {
    "sensitive": false,
    "type": [
      "object",
      {
        "kkp-test-pool1": [
          "object",
          {
            "providerSpec": [
              "object",
              {
                "annotations": [
                  "object",
                  {
                    "cluster.k8s.io/cluster-api-autoscaler-node-group-max-size": "string",
                    "cluster.k8s.io/cluster-api-autoscaler-node-group-min-size": "string",
                    "k8c.io/operating-system-profile": "string"
                  }
                ],
                "cloudProviderSpec": [
                  "object",
                  {
                    "allowInsecure": "bool",
                    "cluster": "string",
                    "cpus": "number",
                    "datacenter": "string",
                    "datastore": "string",
                    "datastoreCluster": "string",
                    "diskSizeGB": "number",
                    "folder": "string",
                    "memoryMB": "number",
                    "resourcePool": "string",
                    "templateVMName": "string",
                    "vmNetName": "string"
                  }
                ],
                "operatingSystem": "string",
                "operatingSystemSpec": [
                  "object",
                  {
                    "distUpgradeOnBoot": "bool"
                  }
                ],
                "sshPublicKeys": [
                  "tuple",
                  [
                    "string"
                  ]
                ]
              }
            ],
            "replicas": "number"
          }
        ]
      }
    ],
    "value": {
      "kkp-test-pool1": {
        "providerSpec": {
          "annotations": {
            "cluster.k8s.io/cluster-api-autoscaler-node-group-max-size": "2",
            "cluster.k8s.io/cluster-api-autoscaler-node-group-min-size": "2",
            "k8c.io/operating-system-profile": ""
          },
          "cloudProviderSpec": {
            "allowInsecure": true,
            "cluster": "CLUSTER",
            "cpus": 2,
            "datacenter": "DATACENTER",
            "datastore": "DATASTORE",
            "datastoreCluster": "",
            "diskSizeGB": 10,
            "folder": "/Customers/TEST/kubermatic/kubeone",
            "memoryMB": 2048,
            "resourcePool": "Test_pool",
            "templateVMName": "ubuntu-22.04-server-cloudimg-kubeone-amd64",
            "vmNetName": "NETWORK"
          },
          "operatingSystem": "ubuntu",
          "operatingSystemSpec": {
            "distUpgradeOnBoot": false
          },
          "sshPublicKeys": [
            "ecdsa-sha2-nistp521 <REDACTED> kubeone"
          ]
        },
        "replicas": 2
      }
    }
  }
}

When I run kubeone apply -m kubeone.yml -t tf.json -c credentials.yml I get the following error message at the step Creating worker machines...:

WARN[10:30:51 CET] Task failed, error was: kubernetes: creating *v1alpha1.MachineDeployment kube-system/kkp-test-pool1
admission webhook "machinedeployments.machine-controller.kubermatic.io" denied the request: validation failed: failed to get folder "/Customers/TEST/kubermatic/kubeone": folder '/Customers/TEST/kubermatic/kubeone' not found
Expected behavior

I expect that the worker nodes will be created in the specified vSphere folder.

How to reproduce the issue?

Setup the KubeOne VMs via Terraform and use the following value in the terraform.tfvars file:

folder_name = "/Customers/TEST/kubermatic/kubeone"

What KubeOne version are you using?
$ kubeone version
{
  "kubeone": {
    "major": "1",
    "minor": "6",
    "gitVersion": "1.6.0",
    "gitCommit": "8b0973ca77856dca920798bbd5ff6b5c0f3f4856",
    "gitTreeState": "",
    "buildDate": "2023-02-23T19:25:26Z",
    "goVersion": "go1.19.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "56",
    "gitVersion": "v1.56.0",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}
Provide your KubeOneCluster manifest here (if applicable)
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
  kubernetes: '1.24.8'
cloudProvider:
  vsphere: {}
  cloudConfig: |
    [Global]
    secret-name = "vsphere-ccm-credentials"
    secret-namespace = "kube-system"
    port = "443"
    insecure-flag = "1"

    [VirtualCenter "VCENTER"]

    [Workspace]
    server = "SERVER"
    datacenter = "DATACENTER"
    default-datastore="DATASTORE"
    resourcepool-path="Test_pool"
    folder = "kubeone"

    [Disk]
    scsicontrollertype = pvscsi

    [Network]
    public-network = "NETWORK"
What cloud provider are you running on?

VMware vSphere

What operating system are you running in your cluster?

Ubuntu 22.04

Additional information

If I update the value of the key kubeone_workers.value.kkp-test-pool1.cloudProviderSpec.folder in the file tf.json to /DATACENTER/vm/Customers/TEST/kubermatic/kubeone the creation of the worker nodes is working.

I tried to setup the full path for the folder as the value in the terraform.tfvars file (folder_name = "/DATACENTER/vm/Customers/TEST/kubermatic/kubeone").
But with this configuration it fails directly at the Terraform run with the following message:

vsphere_virtual_machine.control_plane[1]: Creating...
╷
│ Error: folder '/DATACENTER/vm/DATACENTER/vm/Customers/TEST/kubermatic/kubeone' not found
│
│   with vsphere_virtual_machine.control_plane[1],
│   on main.tf line 152, in resource "vsphere_virtual_machine" "control_plane":
│  152: resource "vsphere_virtual_machine" "control_plane" {

For me it looks like that the full folder path should be used as value for the key kubeone_workers.value.kkp-test-pool1.cloudProviderSpec.folder in the tf.json file.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the machine-controller vSphere folder handling behind the admission webhook, using the folder values shown in tf.json, terraform.tfvars, and the KubeOne manifest. Reproduce the failure with kubeone apply and compare the accepted and rejected paths; done means the specified folder is resolved consistently and worker creation succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.