apache / apache/airflow

SQLExecuteQueryOperator not timing out within expected timeframe

Open
#45,930 16 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:common-sql
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow Provider(s)

common-sql

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.0.0
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-mysql==5.7.3

### Apache Airflow version

2.10.3

### Operating System

Amazon Linux 2023

### Deployment

Amazon (AWS) MWAA

### Deployment details

The issue happens on a production Amazon (AWS) MWAA environment, and is also reproducible locally using the [MWAA local runner](https://github.com/aws/aws-mwaa-local-runner) (deployed on a Mac).

### What happened

The `SQLExecuteQueryOperator` fails with timeout, but only _after_ the query has completed.

This is the workflow currently happening: `T0: Start -> T1: Run query -> T2: Expected timeout failure -> T3: Query finishes -> T4: Operator fails on timeout`

### What you think should happen instead

Once the execution timeout is reached, the `SQLExecuteQueryOperator` should kill the query and then fail.

This is the expected workflow: `T0: Start -> T1: Run query -> T2: Expected timeout, operator kills query and fails`

### How to reproduce

The issue is reproducible both on a production Amazon (AWS) MWAA environment, and locally using the [MWAA local runner](https://github.com/aws/aws-mwaa-local-runner) (deployed on a Mac).

The DAG below contains two tasks:
1. The first one executes a long-running query on a MySQL database using the `SQLExecuteQueryOperator`.
2. The second one executes a long-running process using the `PythonOperator`.

The `PythonOperator` fails its execution exactly after one minute (as expected), the `SQLExecuteQueryOperator` takes a few minutes until the query completes (not as expected).

```
import os
import time

from datetime import timedelta

from airflow.decorators import dag
from airflow.models.connection import Connection
from airflow.operators.python import PythonOperator
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

@dag(schedule=None)
def test_sql_operator_timeout_dag():

# SQL operator test
connection = Connection(
conn_id="mysql_connection",
conn_type="mysql",
host="xyz",
port="3306",
schema="test",
login="xyz",
password="xyz",
)

env_key = f"AIRFLOW_CONN_{connection.conn_id.upper()}"
connection_uri = connection.get_uri()
os.environ[env_key] = connection_uri

SQLExecuteQueryOperator(
task_id="mysql_long_query",
conn_id=connection.conn_id,
sql="select benchmark(1200000000, md5('when will it end?'));",
execution_timeout=timedelta(minutes=1),
)

# Python operator test
def sleeping_function():
time.sleep(120)

PythonOperator(
task_id="python_long_task",
python_callable=sleeping_function,
execution_timeout=timedelta(minutes=1),
)

test_sql_operator_timeout_dag = test_sql_operator_timeout_dag()
```

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the SQLExecuteQueryOperator and its execution_timeout handling, then compare it with the PythonOperator behavior shown in the reproduction DAG. Reproduce the one-minute timeout using the provided MySQL benchmark query and MWAA local runner details. Done means the SQL query is stopped and the task fails when the timeout is reached, rather than after the query completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
data-engineering, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.