Azure / Azure/azure-cli-extensions

az connectedk8s connect does not support kubeconfig that requires extensions

Open
#8,788 1 comment 0 reactions 0 assignees View on GitHub
Auto-Assign bug Connected Kubernetes customer-reported Service Attention
Dominant language
Python
Stars
454
Forks
1.7k
Avg merge
2d 19h
Merged PRs (30d)
64

Description

### Describe the bug

Gardener exposes a kubeconfig like this:

```yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: removed
extensions:
- extension:
gardenClusterIdentity: leafcloud-production
shootRef:
name: genai
namespace: garden-gnh8hcnimj
name: client.authentication.k8s.io/exec
server: removed
name: garden-gnh8hcnimj--genai-external
contexts:
- context:
cluster: garden-gnh8hcnimj--genai-external
namespace: default
user: garden-gnh8hcnimj--genai
name: garden-gnh8hcnimj--genai-external
current-context: garden-gnh8hcnimj--genai-external
kind: Config
preferences: {}
users:
- name: garden-gnh8hcnimj--genai
user:
exec:
apiVersion: client.authentication.k8s.io/v1
args:
- get-client-certificate
command: kubectl-gardenlogin
env: null
installHint: Follow the instructions on https://github.com/gardener/gardenlogin#installation
to install gardenlogin
interactiveMode: IfAvailable
provideClusterInfo: true
```

For this, the `kubectl-gardenlogin get-client-certificate` command needs to be provided with the `extension` attributes in order to correctly get the token.

The current version of the `kubernetes` package does not support 1) the provideClusterInfo properties, and 2) the additional support for the extension properties.

The following fixes have already been implemented on the Kubernetes client side:

- https://github.com/kubernetes-client/python/issues/2298

One more fix awaiting there to support the extension properties:

- https://github.com/kubernetes-client/python/pull/2396

Upgrading the Kubernetes package to the latest (`kubernetes==32.0.1`) and manually patching the run command of the Exec Provider like this fixes the issue (in `custom.py`):

```python
def run(self, previous_response=None):
import sys
import subprocess
from kubernetes.config.config_exception import ConfigException
is_interactive = hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()
kubernetes_exec_info = {
'apiVersion': self.api_version,
'kind': 'ExecCredential',
'spec': {
'interactive': is_interactive
}
}
if previous_response:
kubernetes_exec_info['spec']['response'] = previous_response
if self.cluster:
kubernetes_exec_info['spec']['cluster'] = self.cluster.value
for extension in self.cluster.value["extensions"]:
if extension["name"] == "client.authentication.k8s.io/exec":
kubernetes_exec_info["spec"]["cluster"]["config"] = extension["extension"]
break

self.env['KUBERNETES_EXEC_INFO'] = json.dumps(kubernetes_exec_info)
process = subprocess.Popen(
self.args,
stdout=subprocess.PIPE,
stderr=sys.stderr if is_interactive else subprocess.PIPE,
stdin=sys.stdin if is_interactive else None,
cwd=self.cwd,
env=self.env,
universal_newlines=True,
shell=self.shell)
(stdout, stderr) = process.communicate()
exit_code = process.wait()
if exit_code != 0:
msg = 'exec: process returned %d' % exit_code
stderr = stderr.strip()
if stderr:
msg += '. %s' % stderr
raise ConfigException(msg)
try:
data = json.loads(stdout)
except ValueError as de:
raise ConfigException(
'exec: failed to decode process output: %s' % de)
for key in ('apiVersion', 'kind', 'status'):
if key not in data:
raise ConfigException(
'exec: malformed response. missing key \'%s\'' % key)
if data['apiVersion'] != self.api_version:
raise ConfigException(
'exec: plugin api version %s does not match %s' %
(data['apiVersion'], self.api_version))
return data['status']

from kubernetes.config.exec_provider import ExecProvider

ExecProvider.run = run
```

Applying this fix will significantly help other users connecting their Gardener clusters (for instance, on `leaf.cloud`) as Arc-Enabled Kubernetes Clusters

### Related command

az connectedk8s connect

### Errors

```
'ERROR:root:exec: process returned 1. Error: name must be specified. Hint: update kubectl in case you are using a version older than v1.20.0
Usage:
gardenlogin get-client-certificate [flags
]

Flags:
...
```

### Issue script & Debug output

Relevant debug output:

```Step: 2025-05-24T06-38-31Z: Escape Proxy Settings, if passed in
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /metadata/endpoints?api-version=2022-09-01 HTTP/1.1" 200 1795
root: exec: process returned 1. Error: failed to complete command options: ExecCredential does not contain cluster information
Usage:
gardenlogin get-client-certificate [flags]

Flags:
--access-level string Defines the access level of the credential returned by the plugin. Can be "auto", "admin", or "viewer".
"auto" - Attempts to obtain admin-level credentials. If unsuccessful, it defaults to viewer-level credentials.
"admin" - Returns a credential with cluster-admin privileges.
"viewer" - Returns a credential with read-only access to non-encrypted API resources. (default "auto")
--certificate-cache-dir string Directory of the certificate cache (default "/Users/pnlmw585/.kube/cache/gardenlogin")
--expiration-seconds int Validity duration of the requested credential (default 900)
--garden-cluster-identity string Cluster identity of the garden cluster
-h, --help help for get-client-certificate
--name string Name of the shoot cluster
--namespace string Namespace of the shoot cluster

Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
--config string config file (default is ~/.garden/gardenlogin.yaml)
--log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log-dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log-file string If non-empty, use this log file (no effect when -logtostderr=true)
--log-file-max-size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
--stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

Error: failed to complete command options: ExecCredential does not contain cluster information
Step: 2025-05-24T06-38-31Z: Checking Connectivity to Cluster
kubernetes.client.rest: response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
```

### Expected behavior

The azconnectedk8s using both the Cluster Information and the extension attributes

### Environment Summary

```
az --version
azure-cli 2.71.0 *

core 2.71.0 *
telemetry 1.1.0

Extensions:
connectedk8s 1.10.7

Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1

Python location '/opt/homebrew/Cellar/azure-cli/2.71.0/libexec/bin/python'
Config directory '/Users/mats/.azure'
Extensions directory '/Users/mats/.azure/cliextensions'

Python (Darwin) 3.12.9 (main, Feb 4 2025, 14:38:38) [Clang 16.0.0 (clang-1600.0.26.6)]

Legal docs and information: aka.ms/AzureCliLegal

You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade'
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.