duckdb / duckdb/duckdb-postgres

when write back to GP locked by Postgres scanner

Open
#209 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
372
Forks
105
Avg merge
10h 19m
Merged PRs (30d)
18

Description

### What happens?

I am writing functions in PL/Python in Greenplum (GP) with three main steps:

1. Using DuckDB to read GP tables.
2. Calculating a result table, "rst," in DuckDB, extracting the DDL of the table, and creating the table in GP (need to add distribution keys, compression, etc., which default CREATE does not include,use plpy.execute(ddl)).
```python
tb="gto_skc_store_step_kpi_summary_duck"
sql="""select sql from duckdb_tables() where table_name like 'tmp_rst' and schema_name = 'main';"""
ddl=dd.execute(sql).fetchone()[0]
ddl=f"""drop table if exists tenant_{tenant}_adm.{tb};
"""+ddl.replace('CREATE TABLE tmp_rst', f'CREATE TABLE tenant_{tenant}_adm.{tb}') \
.replace('DOUBLE', 'numeric').replace(';',"""WITH (
appendonly=true,
orientation=column,
compresslevel=6,
compresstype=zlib
)
DISTRIBUTED BY (skc_sk);""")

plpy.notice(ddl)
plpy.execute(ddl)
```
4. Writing the "rst" result table to GP (using COPY or INSERT).
```sql
insert into gp.tenant_{tenant}_adm.{tb}
select * from tmp_rst
-- or
COPY gp.tenant_{tenant}_adm.gto_skc_store_step_kpi_summary_duck FROM '/home/gpadmin/{tb}.parquet';
```

The third step never completes, and checking pg_stat_activity shows a locked state.
However, if split it into two functions, the first completes the first two steps, and write the result to a Parquet file, and the second reads the Parquet file, writing to GP, finishing in just tens of seconds.
Why?
pg_stat_activity:
![image](https://github.com/duckdb/duckdb/assets/25274931/e31aef4d-ab9f-4134-b508-2dd8276cb075)
and pg_locks in following excel:
[pg_locks.xlsx](https://github.com/duckdb/duckdb/files/14929750/pg_locks.xlsx)

### To Reproduce

```python
import duckdb as dd
sql=f"""COPY gp.tenant_{tenant}_adm.gto_skc_store_step_kpi_summary_duck FROM '/home/gpadmin/{tb}.parquet';"""
dd.execute(sql)
```

### OS:

centos7

### DuckDB Version:

0.10.1

### DuckDB Client:

python3.9

### Full Name:

wang cz

### Affiliation:

Linezone

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Yes

### Did you include all code required to reproduce the issue?

- [X] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [X] Yes, I have

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.