Large string logs from trigger are not written to log file intermittently.
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.1.8
### What happened and how to reproduce it?
We noticed reports where large string logs from trigger are not displayed in UI intermittently. The lines are also not present in the log file on the disk. We have custom Spark and Kubernetes based triggers where we download the container log and pod log to be logged using `self.log.info(f"Output : {pod_log}")` inside custom triggers. The pod_log sometimes are large as 4-5 MB as a single string passed through socket. As per our understanding from Airflow 3 the logs pass through sockets and then reach the structlog logger in process_log_messages_from_subprocess compared to Airflow 2 where it directly got written to the file from the trigger through file handler. So while processing large strings and when the triggerer is also busy the messages seem to get lost to be not logged in the triggerer. Sometimes the large string is broken and `msgspec.json.decode(line)` raises exception while decoding causing them to be skipped. This is not consistently reproducible as it depends on the triggerer load and also the large string.
In Airflow 3 is there a way to skip the socket layer and directly write to the trigger.log file for the trigger similar to Airflow 2.
https://github.com/apache/airflow/blob/44b79fe53981bee4c9ccf163acd927be34042774/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L2008-L2047
### What you think should happen instead?
_No response_
### Operating System
_No response_
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with task-sdk/src/airflow/sdk/execution_time/supervisor.py at the linked lines and trace process_log_messages_from_subprocess, focusing on how large trigger messages pass through the socket and are decoded with msgspec.json.decode(line). Reproduce or investigate intermittent loss and determine how trigger.log should reliably contain complete large messages without decode failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100