Azure / Azure/azure-cli-extensions
az connectedk8s connect does not support kubeconfig that requires extensions
- Dominant language
- Python
- Stars
- 454
- Forks
- 1.7k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 64
Description
### Describe the bug
Gardener exposes a kubeconfig like this:
```yaml
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: removed
extensions:
- extension:
gardenClusterIdentity: leafcloud-production
shootRef:
name: genai
namespace: garden-gnh8hcnimj
name: client.authentication.k8s.io/exec
server: removed
name: garden-gnh8hcnimj--genai-external
contexts:
- context:
cluster: garden-gnh8hcnimj--genai-external
namespace: default
user: garden-gnh8hcnimj--genai
name: garden-gnh8hcnimj--genai-external
current-context: garden-gnh8hcnimj--genai-external
kind: Config
preferences: {}
users:
- name: garden-gnh8hcnimj--genai
user:
exec:
apiVersion: client.authentication.k8s.io/v1
args:
- get-client-certificate
command: kubectl-gardenlogin
env: null
installHint: Follow the instructions on https://github.com/gardener/gardenlogin#installation
to install gardenlogin
interactiveMode: IfAvailable
provideClusterInfo: true
```
For this, the `kubectl-gardenlogin get-client-certificate` command needs to be provided with the `extension` attributes in order to correctly get the token.
The current version of the `kubernetes` package does not support 1) the provideClusterInfo properties, and 2) the additional support for the extension properties.
The following fixes have already been implemented on the Kubernetes client side:
- https://github.com/kubernetes-client/python/issues/2298
One more fix awaiting there to support the extension properties:
- https://github.com/kubernetes-client/python/pull/2396
Upgrading the Kubernetes package to the latest (`kubernetes==32.0.1`) and manually patching the run command of the Exec Provider like this fixes the issue (in `custom.py`):
```python
def run(self, previous_response=None):
import sys
import subprocess
from kubernetes.config.config_exception import ConfigException
is_interactive = hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()
kubernetes_exec_info = {
'apiVersion': self.api_version,
'kind': 'ExecCredential',
'spec': {
'interactive': is_interactive
}
}
if previous_response:
kubernetes_exec_info['spec']['response'] = previous_response
if self.cluster:
kubernetes_exec_info['spec']['cluster'] = self.cluster.value
for extension in self.cluster.value["extensions"]:
if extension["name"] == "client.authentication.k8s.io/exec":
kubernetes_exec_info["spec"]["cluster"]["config"] = extension["extension"]
break
self.env['KUBERNETES_EXEC_INFO'] = json.dumps(kubernetes_exec_info)
process = subprocess.Popen(
self.args,
stdout=subprocess.PIPE,
stderr=sys.stderr if is_interactive else subprocess.PIPE,
stdin=sys.stdin if is_interactive else None,
cwd=self.cwd,
env=self.env,
universal_newlines=True,
shell=self.shell)
(stdout, stderr) = process.communicate()
exit_code = process.wait()
if exit_code != 0:
msg = 'exec: process returned %d' % exit_code
stderr = stderr.strip()
if stderr:
msg += '. %s' % stderr
raise ConfigException(msg)
try:
data = json.loads(stdout)
except ValueError as de:
raise ConfigException(
'exec: failed to decode process output: %s' % de)
for key in ('apiVersion', 'kind', 'status'):
if key not in data:
raise ConfigException(
'exec: malformed response. missing key \'%s\'' % key)
if data['apiVersion'] != self.api_version:
raise ConfigException(
'exec: plugin api version %s does not match %s' %
(data['apiVersion'], self.api_version))
return data['status']
from kubernetes.config.exec_provider import ExecProvider
ExecProvider.run = run
```
Applying this fix will significantly help other users connecting their Gardener clusters (for instance, on `leaf.cloud`) as Arc-Enabled Kubernetes Clusters
### Related command
az connectedk8s connect
### Errors
```
'ERROR:root:exec: process returned 1. Error: name must be specified. Hint: update kubectl in case you are using a version older than v1.20.0
Usage:
gardenlogin get-client-certificate [flags
]
Flags:
...
```
### Issue script & Debug output
Relevant debug output:
```Step: 2025-05-24T06-38-31Z: Escape Proxy Settings, if passed in
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /metadata/endpoints?api-version=2022-09-01 HTTP/1.1" 200 1795
root: exec: process returned 1. Error: failed to complete command options: ExecCredential does not contain cluster information
Usage:
gardenlogin get-client-certificate [flags]
Flags:
--access-level string Defines the access level of the credential returned by the plugin. Can be "auto", "admin", or "viewer".
"auto" - Attempts to obtain admin-level credentials. If unsuccessful, it defaults to viewer-level credentials.
"admin" - Returns a credential with cluster-admin privileges.
"viewer" - Returns a credential with read-only access to non-encrypted API resources. (default "auto")
--certificate-cache-dir string Directory of the certificate cache (default "/Users/pnlmw585/.kube/cache/gardenlogin")
--expiration-seconds int Validity duration of the requested credential (default 900)
--garden-cluster-identity string Cluster identity of the garden cluster
-h, --help help for get-client-certificate
--name string Name of the shoot cluster
--namespace string Namespace of the shoot cluster
Global Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
--config string config file (default is ~/.garden/gardenlogin.yaml)
--log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log-dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log-file string If non-empty, use this log file (no effect when -logtostderr=true)
--log-file-max-size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--one-output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
--stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Error: failed to complete command options: ExecCredential does not contain cluster information
Step: 2025-05-24T06-38-31Z: Checking Connectivity to Cluster
kubernetes.client.rest: response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
```
### Expected behavior
The azconnectedk8s using both the Cluster Information and the extension attributes
### Environment Summary
```
az --version
azure-cli 2.71.0 *
core 2.71.0 *
telemetry 1.1.0
Extensions:
connectedk8s 1.10.7
Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1
Python location '/opt/homebrew/Cellar/azure-cli/2.71.0/libexec/bin/python'
Config directory '/Users/mats/.azure'
Extensions directory '/Users/mats/.azure/cliextensions'
Python (Darwin) 3.12.9 (main, Feb 4 2025, 14:38:38) [Clang 16.0.0 (clang-1600.0.26.6)]
Legal docs and information: aka.ms/AzureCliLegal
You have 2 update(s) available. Consider updating your CLI installation with 'az upgrade'
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.