apache / apache/airflow

Large string logs from trigger are not written to log file intermittently.

Open
#66,158 4 comments 0 reactions 0 assignees View on GitHub
area:logging area:Triggerer kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.1.8

### What happened and how to reproduce it?

We noticed reports where large string logs from trigger are not displayed in UI intermittently. The lines are also not present in the log file on the disk. We have custom Spark and Kubernetes based triggers where we download the container log and pod log to be logged using `self.log.info(f"Output : {pod_log}")` inside custom triggers. The pod_log sometimes are large as 4-5 MB as a single string passed through socket. As per our understanding from Airflow 3 the logs pass through sockets and then reach the structlog logger in process_log_messages_from_subprocess compared to Airflow 2 where it directly got written to the file from the trigger through file handler. So while processing large strings and when the triggerer is also busy the messages seem to get lost to be not logged in the triggerer. Sometimes the large string is broken and `msgspec.json.decode(line)` raises exception while decoding causing them to be skipped. This is not consistently reproducible as it depends on the triggerer load and also the large string.

In Airflow 3 is there a way to skip the socket layer and directly write to the trigger.log file for the trigger similar to Airflow 2.

https://github.com/apache/airflow/blob/44b79fe53981bee4c9ccf163acd927be34042774/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2008-L2047

### What you think should happen instead?

_No response_

### Operating System

_No response_

### Deployment

Virtualenv installation

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with task-sdk/src/airflow/sdk/execution_time/supervisor.py at the linked lines and trace process_log_messages_from_subprocess, focusing on how large trigger messages pass through the socket and are decoded with msgspec.json.decode(line). Reproduce or investigate intermittent loss and determine how trigger.log should reliably contain complete large messages without decode failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.