apache / apache/airflow

Raise KubernetesPodOperator do_xcom_push size limit

Open
#15,761 2 comments 1 reaction 0 assignees View on GitHub
kind:feature provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

Currently KubernetesPodOperator will push the file `/airflow/xcom/return.json` as XCom value.
This value is affected by two limits

* The size limit for the xcom value set by the airflow default built-in xcom backend, but you can provide *a custom XCom backend that does not have that limit*, like an backend that uploads values to S3
* The amount of bytes that can be transferred in a single request from kubernetes stream. (or at least that's what I suspect)

I would expect to be able to get the full contents of the file at the custom Xcom backend

**Use case / motivation**

I want to receive the full contents of `/airflow/xcom/return.json` as XCom value into my custom Xcom backend (that will upload to S3).

Otherwise I need to provide a custom logic in my docker image to upload to S3 instead with the following drawback
* no secure way to pass the aws connection credentials from airflow to the kubernetespodoperator
* duplication of code.

**Are you willing to submit a PR?**

**Related Issues**

Contributor guide

Open the contributing guide

Research direction

Start with KubernetesPodOperator and its handling of /airflow/xcom/return.json, then trace how Kubernetes stream transfer interacts with the built-in and custom XCom backends. Confirm whether the transfer imposes a separate size limit. Done means the full file contents can reach a custom backend without being restricted by the current transfer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
data-engineering, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.