citusdata / citusdata/citus

Sporadic intermediate result leak in Circle CI

Open
#3,910 1 comment 0 reactions 0 assignees View on GitHub
regression tests
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

observed insert..select with re-partitioning leak a file in Circle CI:
```
--- /home/circleci/project/src/test/regress/expected/ensure_no_intermediate_data_leak.out.modified 2020-06-12 18:23:30.745763054 +0000
+++ /home/circleci/project/build-11/src/test/regress/results/ensure_no_intermediate_data_leak.out.modified 2020-06-12 18:23:30.749763152 +0000
@@ -19,21 +19,22 @@
SELECT * FROM run_command_on_workers($$
WITH xact_dirs AS (
SELECT pg_ls_dir('base/pgsql_job_cache') dir WHERE citus_version() NOT ILIKE '%windows%'
), result_files AS (
SELECT dir, pg_ls_dir('base/pgsql_job_cache/' || dir) result_file FROM xact_dirs
)
SELECT array_agg((xact_dirs.dir, result_files.result_file)) FROM xact_dirs LEFT OUTER JOIN result_files ON xact_dirs.dir = result_files.dir;
$$) WHERE result <> '';
nodename | nodeport | success | result
-----------+----------+---------+--------------------------------------------------------------------------
-(0 rows)
+ localhost | 57637 | t | {"(10_0_445,repartitioned_results_20149436507_from_13300003_to_1.data)"}
+(1 row)

```

Contributor guide

Open the contributing guide

Research direction

Start with the regression test under src/test/regress/ensure_no_intermediate_data_leak and its expected output. Reproduce the insert..select with re-partitioning in Circle CI or the test environment, then trace the reported repartitioned_results file. Done means the test no longer observes an intermediate result file and the regression output is clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sql
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.