apache / apache/airflow

Configure return value logging for a DecoratedOperator

Open
#19,670 2 comments 0 reactions 0 assignees View on GitHub
AIP-31 kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

Continuation of #19378

https://github.com/apache/airflow/pull/19378#issuecomment-966898703
> If the user uses `DecoratedOperator` as shown below through [TaskFlow API](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html), there seems to be no way to block XCom return value logs.
>
> ```python
> @task
> def my_task(): # Guess it is `_PythonDecoratedOperator`
> # ... process
> return large_data
>
>
> with DAG(...) as dag:
> data = my_task() # It will make a huge log messages on Airflow UI
> my_second_task(data)
> ```
>
> so, I think additional work will be required in a separate PR for TaskFlow API users.

https://github.com/apache/airflow/pull/19378#issuecomment-967047503
>
>
> Yes we should do something for this but separately. Maybe something like (just a random idea)
>
> ```python
> @task(..., log_return_value="DEBUG")
> def my_task():
> ...
> return large_data
> ```

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reading the referenced TaskFlow API example and the discussion in pull request #19378 about DecoratedOperator return-value logging. Determine how a user should configure logging for the decorated task, including the proposed log_return_value option, and define done as preventing large return values from appearing at the default log level while allowing intentional logging.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.