KubernetesExecutor scheduler crashes with PicklingError on pod_override with kubernetes client 36.x
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow Provider(s)
cncf-kubernetes
### What happened
With `KubernetesExecutor` running **in-cluster**, any task that sets a `V1Pod` `pod_override` in `executor_config` crashes the **scheduler** in a loop:
```
_pickle.PicklingError: Can't pickle local object
._refresh_api_key at 0x...>
when serializing kubernetes.client.configuration.Configuration ...
when serializing dict item 'pod_override' ... ExecuteTask ...
File ".../executors/kubernetes_executor.py", line 230, in execute_async
self.task_queue.put(KubernetesJob(key, command, kube_executor_config, pod_template_file))
```
`execute_async` puts the `KubernetesJob` (embedding the task's `pod_override` `V1Pod`) onto a `multiprocessing.Manager` queue, whose `.put()` pickles synchronously in the scheduler process — so a pickle failure takes the scheduler down rather than failing the task.
### Root cause
In kubernetes-client/python **36.0.0**, the generated model `__init__` changed how the default `local_vars_configuration` is set (PR [kubernetes-client/python#2532](https://github.com/kubernetes-client/python/pull/2532), upgrading to OpenAPI Generator v6.6.0 — [OpenAPITools/openapi-generator#8500](https://github.com/OpenAPITools/openapi-generator/pull/8500)):
```python
# kubernetes 35.x
if local_vars_configuration is None:
local_vars_configuration = Configuration() # empty -> picklable
# kubernetes 36.x
if local_vars_configuration is None:
local_vars_configuration = Configuration.get_default_copy() # copy of the in-cluster default
```
In-cluster, the default `Configuration` carries `refresh_api_key_hook = InClusterConfigLoader._set_config.._refresh_api_key`, a **local closure** that `pickle` cannot serialize. So in 36.x every in-cluster `V1Pod` inherits an unpicklable `Configuration`.
### How to reproduce
In-cluster `KubernetesExecutor`, kubernetes client 36.x, run a task with a `V1Pod` `pod_override`:
```python
from kubernetes.client import models as k8s
PythonOperator(
task_id="t",
python_callable=lambda: None,
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={"cpu": "100m", "memory": "384Mi"},
),
)
]
),
)
},
)
```
### Operating System
Any (Linux, in-cluster). Reproduced on Astro Runtime with Airflow 3.2.x and 3.3.x.
### Versions of Apache Airflow Providers
`apache-airflow-providers-cncf-kubernetes==10.18.0`, `kubernetes==36.0.0` / `36.0.2`, in-cluster `KubernetesExecutor`.
### Deployment
Official Apache Airflow Helm Chart / Astronomer
### Deployment details
In-cluster `KubernetesExecutor` (scheduler loads config via `load_incluster_config`).
### Proposed fix
#68819 — serialize the `pod_override` to a plain dict before queuing (dropping the `Configuration`) and rebuild the `V1Pod` worker-side, so the executor is robust regardless of the kubernetes client version. The worker side already sanitizes pods with `sanitize_for_serialization`; this closes the same gap on the scheduler -> queue boundary. The underlying client behavior should also be reported upstream to kubernetes-client/python (root cause: #2532 / openapi-generator#8500).
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR! (#68819)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @vatsrahul1001 before posting
Contributor guide
Research direction
Start in executors/kubernetes_executor.py at execute_async, then inspect the worker-side sanitize_for_serialization path and the proposed fix in #68819. Done means an in-cluster KubernetesExecutor can queue a task with a V1Pod pod_override under kubernetes-client 36.x without the scheduler crashing on pickle serialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- data-engineering, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100