apache / apache/airflow

Large XCom Payload Causes Task Heartbeat Timeout

Open
#64,628 0 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug needs-triage priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.1.7

### What happened and how to reproduce it?

We attempted to push a very large XCom payload (over 300 MB) from a worker task to the XCom table.
We understand this is not the ideal approach for data of this size, and that other mechanisms are generally better suited.

However, the main reason for opening this issue is the behavior we observed: uploading the XCom value through the supervisor took about 18 minutes. During that time, the supervisor was blocked by the XCom push and could not process heartbeats. As a result, the task timed out and was marked as failed.

This may also affect other backends. The default task timeout is 300 seconds, so if the XCom push takes longer than that, the scheduler marks the task as failed.

Has anyone experienced the same issue, and do you have suggestions for how to solve it?

### What you think should happen instead?

Pushing an XCom value should not block task heartbeats.
If the payload is large and upload takes longer, the task should continue sending heartbeats (or fail with a clear XCom-size error) instead of being marked failed due to heartbeat timeout.

### Operating System

_No response_

### Versions of Apache Airflow Providers

_No response_

### Deployment

None

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the worker supervisor's XCom push and heartbeat path, then inspect how the scheduler handles the default 300-second timeout. Done means a large upload no longer causes heartbeat failure, or it produces a clear XCom-size error instead of marking the task failed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.