apache / apache/airflow

Restore kubernetes-client 36.x support in cncf.kubernetes and google providers (401 auth regression)

Open
#69,023 4 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug provider:cncf-kubernetes provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Context

`apache-airflow-providers-cncf-kubernetes` 10.18.0 widened the kubernetes client bound to allow 36.x (#68041). Client 36.x renamed the bearer-token auth key, which breaks the **google (GKE) provider's synchronous** credentials path with a **401 Unauthorized**. The bound is capped back to `<36.0.0` (35.x, last known-good) as a workaround in #69025 until the provider auth paths support 36.x.

### Root cause (verified against kubernetes-client/python 36.x)

kubernetes-client/python 36.0.0 changed the generated `Configuration` auth handling (kubernetes-client/python#2582): the bearer scheme key was renamed from `authorization` to `BearerToken`. In 36.x:

```python
def auth_settings(self):
auth = {}
if 'BearerToken' in self.api_key or 'authorization' in self.api_key:
auth['BearerToken'] = {
'type': 'api_key', 'in': 'header', 'key': 'authorization',
'value': self.get_api_key_with_prefix('BearerToken', alias='authorization'),
}
return auth

def get_api_key_with_prefix(self, identifier, alias=None):
if self.refresh_api_key_hook is not None:
self.refresh_api_key_hook(self)
key = self.api_key.get(identifier, self.api_key.get(alias) if alias is not None else None)
if key:
prefix = self.api_key_prefix.get(identifier) # looked up by 'BearerToken', NOT the alias
...
```

36.0.1 (kubernetes-client/python#2585) fixed this **only** for `load_incluster_config()` / `load_kube_config()` (sync + async) by writing the token under the new `BearerToken` key. It did **not** fix code that hand-builds a `Configuration`.

The google GKE **sync** hook hand-builds the config and registers the bearer token/prefix under the **old** `authorization` key (`GKEKubernetesHook.get_conn` / `_get_config`, `providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98`):

```python
client.Configuration(
api_key_prefix={"authorization": "Bearer"},
api_key={"authorization": token},
)
configuration.refresh_api_key_hook = self._refresh_api_key_hook # also sets api_key["authorization"]
```

On 36.x, `get_api_key_with_prefix('BearerToken', alias='authorization')` finds the **token** via the `authorization` alias, but looks up the **prefix** by the primary identifier `BearerToken` — which is absent — so the `Bearer ` prefix is dropped. The request goes out as `Authorization: ` instead of `Authorization: Bearer `, and the API server rejects it with **401**. On 35.x the scheme used identifier `authorization` for both the key and the prefix, so it worked. This is why downgrading the provider to 10.17.1 (client 35.x) fixes it, and why the failure is **immediate** (not token-rotation related).

The **async** GKE path is unaffected — it sets the header directly via `ApiClient(header_name=..., header_value="Bearer ")`, bypassing `auth_settings()`. cncf.kubernetes's own hooks go through `load_incluster_config`/`load_kube_config`, which 36.0.1 already fixed.

The same 36.x `Configuration` change is also behind the in-cluster `pod_override` `PicklingError` (#68827), partially addressed by #68848.

### Follow-up work to lift the cap

- [ ] **google GKE sync hook** — register the bearer token/prefix under `BearerToken` (keeping `authorization` for client-35.x back-compat), or set the `Authorization` header directly on the `ApiClient` as the async path already does (`kubernetes_engine.py:75-98`).
- [ ] **Audit** any other provider code that hand-builds a kubernetes `Configuration` with `api_key={'authorization': ...}` and apply the same fix.
- [ ] **Regression test** asserting the outgoing request carries `Authorization: Bearer ` on client 36.x (it must fail on the pre-fix code).

### Acceptance criteria

- google (GKE) and cncf.kubernetes validated on client 36.x against real clusters with a confirmed `Authorization: Bearer …` header.
- Cap (#69025) lifted to allow 36.x again.

### References

- kubernetes-client/python#2582 (auth_settings BearerToken key rename / 401), kubernetes-client/python#2585 (36.0.1 loader-only fix)
- #68041 (the widening), #68827 + #68848 (pickling), #69025 (the cap), 10.17.1 (last known-good)

---
Drafted-by: Claude Code (Opus 4.8) (no human review before posting)

Contributor guide

Open the contributing guide

Research direction

Start with GKEKubernetesHook.get_conn and _get_config in providers/google/src/airflow/providers/google/cloud/hooks/kubernetes_engine.py:75-98, comparing the synchronous configuration with the async path. Audit other provider code that hand-builds Kubernetes Configuration objects, then add the named regression coverage and verify that client 36.x sends an Authorization: Bearer header; done includes lifting cap #69025 after provider validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, kubernetes, python
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.