hydradatabase / hydradatabase/columnar

[Bug]: The pg_terminate_backend could not temrinate the running query

Open
#283 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
3k
Forks
106
PR merge metrics
No merged PRs in 30d

Description

### What's wrong?

We create a large partition table, which structure like

A
Partition key: LIST (customer_id)

A_c_319
Partition of: A FOR VALUES IN (319)
Partition key: RANGE (fill_dt)
Partition constraint: ((customer_id IS NOT NULL) AND (customer_id = 319))

A_c_319_203211
Partition of: A_c_319 FOR VALUES FROM ('2005-01-01') TO ('2023-01-01')
Partition key: HASH (claim_id)

A_c_319_202212_p0
Partition of: A_c_319_202212 FOR VALUES WITH (modulus 16, remainder 0)
Access method: columnar
......

We have total 16601 columnar leaf tables, and the table has more than 1100 columns.

Now, when we run aggregation query, it will sooner exhausted 180GB RAM (except the shared_buffer fix size 120GB), and fully use the 340GB swap space, and finally trigger OOM

We try to run pg_terminate_backend() or pg_cancel_backend(), they do not work.
If we try to kill the query process with kill -9, it crash the DB instance immediately

When it lost control, we check the process Signal stauts, it is like

cat /proc/3496124/status|grep -i SIG

SigQ: 2/1286654
SigPnd: 0000000000000000
SigBlk: 0000000000400000
SigIgn: 0000000001301800
SigCgt: 0000000020006287

The wchan value , at the begin of query run is
rq_qos_wait, then, for a period
0, then, after the query start to use swap
__folio_lock_or_retry

We keep running sript which loop try to issue pg_terminate_backend command, sometime, it can terminate the query process after thousands try, sometime, it will keep trying until OOM occurred.

Is there any way to wake up the sleep process to response the termination signal?? We have 20 mounted VMware disks, and many of them are always saturated, the backend is SAN storage.

Thanks a lot
Frank

Contributor guide

No contributing guide indexed for this repository

Research direction

The report names no source files or tests; its entry points are pg_terminate_backend(), pg_cancel_backend(), and the backend process state in /proc/.../status. Start by reproducing the partitioned columnar aggregation under memory and swap pressure, then trace termination handling while the process waits in rq_qos_wait or __folio_lock_or_retry. Done means termination reliably interrupts the query without killing the database instance.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.