apache / apache/airflow

limit print log of SQL statement

Open
#54,430 3 comments 0 reactions 0 assignees View on GitHub
area:logging kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

The print log _self.log.info("Running statement: %s, parameters: %s", sql_statement, parameters)_ case the UI to lag is a case where the sql statement is long e.g insert statement with 150000 row
https://github.com/apache/airflow/blob/569599ab4fc6a3c7b6e34e23bfea65bb50551c1f/providers/common/sql/src/airflow/providers/common/sql/hooks/sql.py#L810C68-L810C81

**Feature Idea**: Reduce the sql_statement string length to 1000 characters or something not to large

### Use case/motivation

When **sql_statement** is large, like an insert statement with 1 million rows, the UI will not lag when viewing logs and also this will reduce the size of the log file

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in providers/common/sql/src/airflow/providers/common/sql/hooks/sql.py at the SQL statement logging call identified in the issue. Determine how to cap the logged SQL text while preserving useful parameter and statement context. Done means very large statements no longer produce unbounded log output, while ordinary statements remain readable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.