duckdb / duckdb/duckdb-postgres
when write back to GP locked by Postgres scanner
- Dominant language
- C++
- Stars
- 372
- Forks
- 105
- Avg merge
- 10h 19m
- Merged PRs (30d)
- 18
Description
### What happens?
I am writing functions in PL/Python in Greenplum (GP) with three main steps:
1. Using DuckDB to read GP tables.
2. Calculating a result table, "rst," in DuckDB, extracting the DDL of the table, and creating the table in GP (need to add distribution keys, compression, etc., which default CREATE does not include,use plpy.execute(ddl)).
```python
tb="gto_skc_store_step_kpi_summary_duck"
sql="""select sql from duckdb_tables() where table_name like 'tmp_rst' and schema_name = 'main';"""
ddl=dd.execute(sql).fetchone()[0]
ddl=f"""drop table if exists tenant_{tenant}_adm.{tb};
"""+ddl.replace('CREATE TABLE tmp_rst', f'CREATE TABLE tenant_{tenant}_adm.{tb}') \
.replace('DOUBLE', 'numeric').replace(';',"""WITH (
appendonly=true,
orientation=column,
compresslevel=6,
compresstype=zlib
)
DISTRIBUTED BY (skc_sk);""")
plpy.notice(ddl)
plpy.execute(ddl)
```
4. Writing the "rst" result table to GP (using COPY or INSERT).
```sql
insert into gp.tenant_{tenant}_adm.{tb}
select * from tmp_rst
-- or
COPY gp.tenant_{tenant}_adm.gto_skc_store_step_kpi_summary_duck FROM '/home/gpadmin/{tb}.parquet';
```
The third step never completes, and checking pg_stat_activity shows a locked state.
However, if split it into two functions, the first completes the first two steps, and write the result to a Parquet file, and the second reads the Parquet file, writing to GP, finishing in just tens of seconds.
Why?
pg_stat_activity:

and pg_locks in following excel:
[pg_locks.xlsx](https://github.com/duckdb/duckdb/files/14929750/pg_locks.xlsx)
### To Reproduce
```python
import duckdb as dd
sql=f"""COPY gp.tenant_{tenant}_adm.gto_skc_store_step_kpi_summary_duck FROM '/home/gpadmin/{tb}.parquet';"""
dd.execute(sql)
```
### OS:
centos7
### DuckDB Version:
0.10.1
### DuckDB Client:
python3.9
### Full Name:
wang cz
### Affiliation:
Linezone
### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
### Did you include all relevant data sets for reproducing the issue?
Yes
### Did you include all code required to reproduce the issue?
- [X] Yes, I have
### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?
- [X] Yes, I have
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.