Raise KubernetesPodOperator do_xcom_push size limit
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
**Description**
Currently KubernetesPodOperator will push the file `/airflow/xcom/return.json` as XCom value.
This value is affected by two limits
* The size limit for the xcom value set by the airflow default built-in xcom backend, but you can provide *a custom XCom backend that does not have that limit*, like an backend that uploads values to S3
* The amount of bytes that can be transferred in a single request from kubernetes stream. (or at least that's what I suspect)
I would expect to be able to get the full contents of the file at the custom Xcom backend
**Use case / motivation**
I want to receive the full contents of `/airflow/xcom/return.json` as XCom value into my custom Xcom backend (that will upload to S3).
Otherwise I need to provide a custom logic in my docker image to upload to S3 instead with the following drawback
* no secure way to pass the aws connection credentials from airflow to the kubernetespodoperator
* duplication of code.
**Are you willing to submit a PR?**
**Related Issues**
Contributor guide
Research direction
Start with KubernetesPodOperator and its handling of /airflow/xcom/return.json, then trace how Kubernetes stream transfer interacts with the built-in and custom XCom backends. Confirm whether the transfer imposes a separate size limit. Done means the full file contents can reach a custom backend without being restricted by the current transfer behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- data-engineering, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100