citusdata / citusdata/citus

Repartition joins are not compatible with citus.force_max_query_parallelization

Open
#5,345 0 comments 0 reactions 0 assignees View on GitHub
bug tool_reported
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

`readyTaskCount` is typically too high for repartition joins: https://github.com/citusdata/citus/blob/5f46372416f26cd9992fbd49e79fcfdfcbeaa5eb/src/backend/distributed/executor/adaptive_executor.c#L2742-L2752

```SQL
-- create a table and generate 10K rows
create table test(a int, b int);
insert into test select i % 10, i % 15 from generate_series(0,10000) i;
SELECT create_distributed_table('test', 'a');

-- joining two 10K tables results in 6M rows
select count(*) from test t1, test t2 WHERE t1.a = t2.b;
count
---------
6670667
(1 row)

Time: 4116.436 ms (00:04.116)

set citus.force_max_query_parallelization to on;
WARNING: could not issue cancel request
DETAIL: Client error: PQcancel() -- socket() failed: Too many open files

ERROR: kqueue failed: Too many open files
Time: 1164.361 ms (00:01.164)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.