matrixorigin / matrixorigin/matrixone

[Bug]: Dev AP q1811 exceeds configured 7200s SQL timeout without cancellation

Open
#28,007 2 comments 0 reactions 1 assignee Claimed by @aptend View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

In the dev AP regression workload, `@q1811_adv_range_frame_wsl` continues running beyond the configured 7200-second SQL timeout. The runner does not emit a timeout result or advance to the next case.

## Environment

- Workflow branch: `codex/mo-ap-regression-dev-assets`
- Workflow commit: `138640f241f17d28910c366324ab31893ae9ec56`
- Deployment: dev freetier cluster with the OSS-loaded `jinpan_001` dataset
- MatrixOne build SHA: not emitted by this workflow; needs confirmation during triage
- Date: 2026-09-02

## Steps to reproduce

1. Dispatch `MO AP Regression (Dev)` with `test_phase=all`, `data_mode=fresh`.
2. Allow the advanced join shard 0 to execute `tools/mo-regression-test/cases/jinpan_001/join_adv/j_batch_041.sql`.
3. Observe `@q1811_adv_range_frame_wsl`, which performs a window aggregate over `dwd_dcp.dwd_s4_acdoca` and a left join to `dwd_dcp.dwd_bw_ztbpc002_com`.

The workflow invokes this case with `--sql-timeout 7200`.

## Actual behavior

- The previous `@q1810_adv_invoice_having_avg` completed in 71,544.40 ms at `2026-09-02 14:25:08 +08:00`.
- `@q1811_adv_range_frame_wsl` then started at the same time and was still running at `2026-09-02 18:11 +08:00`, over 3 hours 46 minutes later.
- No `cost:` line, SQL error, or `Execution timeout after 7200s` line was emitted for q1811.
- The Advanced Join job therefore remains active and blocks the remaining serialized shards.

## Expected behavior

The query should either complete, return a MatrixOne error, or be canceled and reported by the regression runner no later than the configured 7200-second SQL timeout. A single SQL statement must not keep the regression shard running past its configured timeout.

## Stability and controls

- Reproducer: 1/1 on the current dev workload.
- Control: q1810 in the same connection completed successfully immediately before q1811.
- Scope: this report establishes the timeout/hang symptom on dev only. Reproduction on the latest official `main` has not yet been performed.

## Evidence

- Failing active job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33482459227/job/99890422550
- Case source: `tools/mo-regression-test/cases/jinpan_001/join_adv/j_batch_041.sql`, `@q1811_adv_range_frame_wsl`

## Code analysis

The regression runner wraps `cursor.execute()` and `cursor.fetchall()` with `func_timeout(7200, ...)` and configures PyMySQL `read_timeout`/`write_timeout` to the same value. The observed duration exceeds both values. It is not yet confirmed whether the root cause is a MatrixOne query that does not return/cancel, or a client-side timeout that cannot interrupt this operation.

## Regression coverage

After root cause is fixed, add a bounded big-data/AP regression that verifies a long-running query exits at the configured SQL timeout and the runner continues with the next case.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.