apache / apache/airflow

KubernetesExecutor scheduler crashes with PicklingError on pod_override with kubernetes client 36.x

Open
#68,827 0 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow Provider(s)

cncf-kubernetes

### What happened

With `KubernetesExecutor` running **in-cluster**, any task that sets a `V1Pod` `pod_override` in `executor_config` crashes the **scheduler** in a loop:

```
_pickle.PicklingError: Can't pickle local object
._refresh_api_key at 0x...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override' ... ExecuteTask ...
File ".../executors/kubernetes_executor.py", line 230, in execute_async
self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))
```

`execute_async` puts the `KubernetesJob` (embedding the task's `pod_override` `V1Pod`) onto a `multiprocessing.Manager` queue, whose `.put()` pickles synchronously in the scheduler process — so a pickle failure takes the scheduler down rather than failing the task.

### Root cause

In kubernetes-client/python **36.0.0**, the generated model `__init__` changed how the default `local_vars_configuration` is set (PR [kubernetes-client/python#2532](https://github.com/kubernetes-client/python/pull/2532), upgrading to OpenAPI Generator v6.6.0 — [OpenAPITools/openapi-generator#8500](https://github.com/OpenAPITools/openapi-generator/pull/8500)):

```python
# kubernetes 35.x
if local_vars_configuration is None:
local_vars_configuration = Configuration() # empty -> picklable

# kubernetes 36.x
if local_vars_configuration is None:
local_vars_configuration = Configuration.get_default_copy() # copy of the in-cluster default
```

In-cluster, the default `Configuration` carries `refresh_api_key_hook = InClusterConfigLoader._set_config.._refresh_api_key`, a **local closure** that `pickle` cannot serialize. So in 36.x every in-cluster `V1Pod` inherits an unpicklable `Configuration`.

### How to reproduce

In-cluster `KubernetesExecutor`, kubernetes client 36.x, run a task with a `V1Pod` `pod_override`:

```python
from kubernetes.client import models as k8s

PythonOperator(
task_id="t",
python_callable=lambda: None,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={"cpu": "100m", "memory": "384Mi"},
),
)
]
),
)
},
)
```

### Operating System

Any (Linux, in-cluster). Reproduced on Astro Runtime with Airflow 3.2.x and 3.3.x.

### Versions of Apache Airflow Providers

`apache-airflow-providers-cncf-kubernetes==10.18.0`, `kubernetes==36.0.0` / `36.0.2`, in-cluster `KubernetesExecutor`.

### Deployment

Official Apache Airflow Helm Chart / Astronomer

### Deployment details

In-cluster `KubernetesExecutor` (scheduler loads config via `load_incluster_config`).

### Proposed fix

#68819 — serialize the `pod_override` to a plain dict before queuing (dropping the `Configuration`) and rebuild the `V1Pod` worker-side, so the executor is robust regardless of the kubernetes client version. The worker side already sanitizes pods with `sanitize_for_serialization`; this closes the same gap on the scheduler -> queue boundary. The underlying client behavior should also be reported upstream to kubernetes-client/python (root cause: #2532 / openapi-generator#8500).

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR! (#68819)

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

---
Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting

Contributor guide

Open the contributing guide

Research direction

Start in executors/kubernetes_executor.py at execute_async, then inspect the worker-side sanitize_for_serialization path and the proposed fix in #68819. Done means an in-cluster KubernetesExecutor can queue a task with a V1Pod pod_override under kubernetes-client 36.x without the scheduler crashing on pickle serialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
data-engineering, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.