Configure return value logging for a DecoratedOperator
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
Continuation of #19378
https://github.com/apache/airflow/pull/19378#issuecomment-966898703
> If the user uses `DecoratedOperator` as shown below through [TaskFlow API](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html), there seems to be no way to block XCom return value logs.
>
> ```python
> @task
> def my_task(): # Guess it is `_PythonDecoratedOperator`
> # ... process
> return large_data
>
>
> with DAG(...) as dag:
> data = my_task() # It will make a huge log messages on Airflow UI
> my_second_task(data)
> ```
>
> so, I think additional work will be required in a separate PR for TaskFlow API users.
https://github.com/apache/airflow/pull/19378#issuecomment-967047503
>
>
> Yes we should do something for this but separately. Maybe something like (just a random idea)
>
> ```python
> @task(..., log_return_value="DEBUG")
> def my_task():
> ...
> return large_data
> ```
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by reading the referenced TaskFlow API example and the discussion in pull request #19378 about DecoratedOperator return-value logging. Determine how a user should configure logging for the decorated task, including the proposed log_return_value option, and define done as preventing large return values from appearing at the default log level while allowing intentional logging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100