Restore kubernetes-client 36.x support in cncf.kubernetes and google providers (401 auth regression)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Context
`apache-airflow-providers-cncf-kubernetes` 10.18.0 widened the kubernetes client bound to allow 36.x (#68041). Client 36.x renamed the bearer-token auth key, which breaks the **google (GKE) provider's synchronous** credentials path with a **401 Unauthorized**. The bound is capped back to `<36.0.0` (35.x, last known-good) as a workaround in #69025 until the provider auth paths support 36.x.
### Root cause (verified against kubernetes-client/python 36.x)
kubernetes-client/python 36.0.0 changed the generated `Configuration` auth handling (kubernetes-client/python#2582): the bearer scheme key was renamed from `authorization` to `BearerToken`. In 36.x:
```python
def auth_settings(self):
auth = {}
if 'BearerToken' in self.api_key or 'authorization' in self.api_key:
auth['BearerToken'] = {
'type': 'api_key', 'in': 'header', 'key': 'authorization',
'value': self.get_api_key_with_prefix('BearerToken', alias='authorization'),
}
return auth
def get_api_key_with_prefix(self, identifier, alias=None):
if self.refresh_api_key_hook is not None:
self.refresh_api_key_hook(self)
key = self.api_key.get(identifier, self.api_key.get(alias) if alias is not None else None)
if key:
prefix = self.api_key_prefix.get(identifier) # looked up by 'BearerToken', NOT the alias
...
```
36.0.1 (kubernetes-client/python#2585) fixed this **only** for `load_incluster_config()` / `load_kube_config()` (sync + async) by writing the token under the new `BearerToken` key. It did **not** fix code that hand-builds a `Configuration`.
The google GKE **sync** hook hand-builds the config and registers the bearer token/prefix under the **old** `authorization` key (`GKEKubernetesHook.get_conn` / `_get_config`, `providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98`):
```python
client.Configuration(
api_key_prefix={"authorization": "Bearer"},
api_key={"authorization": token},
)
configuration.refresh_api_key_hook = self._refresh_api_key_hook # also sets api_key["authorization"]
```
On 36.x, `get_api_key_with_prefix('BearerToken', alias='authorization')` finds the **token** via the `authorization` alias, but looks up the **prefix** by the primary identifier `BearerToken` — which is absent — so the `Bearer ` prefix is dropped. The request goes out as `Authorization: ` instead of `Authorization: Bearer `, and the API server rejects it with **401**. On 35.x the scheme used identifier `authorization` for both the key and the prefix, so it worked. This is why downgrading the provider to 10.17.1 (client 35.x) fixes it, and why the failure is **immediate** (not token-rotation related).
The **async** GKE path is unaffected — it sets the header directly via `ApiClient(header_name=..., header_value="Bearer ")`, bypassing `auth_settings()`. cncf.kubernetes's own hooks go through `load_incluster_config`/`load_kube_config`, which 36.0.1 already fixed.
The same 36.x `Configuration` change is also behind the in-cluster `pod_override` `PicklingError` (#68827), partially addressed by #68848.
### Follow-up work to lift the cap
- [ ] **google GKE sync hook** — register the bearer token/prefix under `BearerToken` (keeping `authorization` for client-35.x back-compat), or set the `Authorization` header directly on the `ApiClient` as the async path already does (`kubernetes_engine.py:75-98`).
- [ ] **Audit** any other provider code that hand-builds a kubernetes `Configuration` with `api_key={'authorization': ...}` and apply the same fix.
- [ ] **Regression test** asserting the outgoing request carries `Authorization: Bearer ` on client 36.x (it must fail on the pre-fix code).
### Acceptance criteria
- google (GKE) and cncf.kubernetes validated on client 36.x against real clusters with a confirmed `Authorization: Bearer …` header.
- Cap (#69025) lifted to allow 36.x again.
### References
- kubernetes-client/python#2582 (auth_settings BearerToken key rename / 401), kubernetes-client/python#2585 (36.0.1 loader-only fix)
- #68041 (the widening), #68827 + #68848 (pickling), #69025 (the cap), 10.17.1 (last known-good)
---
Drafted-by: Claude Code (Opus 4.8) (no human review before posting)
Contributor guide
Research direction
Start with GKEKubernetesHook.get_conn and _get_config in providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98, comparing the synchronous configuration with the async path. Audit other provider code that hand-builds Kubernetes Configuration objects, then add the named regression coverage and verify that client 36.x sends an Authorization: Bearer header; done includes lifting cap #69025 after provider validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, kubernetes, python
- Domain
- authentication, backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100