Repartition joins that have JSONB columns may run into issues
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
I'm copy/pasting the following transcript from a user conversation. This issue triggered when one of the large tables had a new `JSON` column. It's worth testing whether repartition joins exhibit any issues with large json fields.
##
During our call this morning I mentioned a failed task error I had been getting when the example join contained a json field. It turns out I had distributed one of the two large trip tables incorrectly, so they were being joined with repartition rather than colocation.
Meaning that this is not a blocker for us, as none of our joins should be re-partition in production.
If you’re curious, here’s the worker log:
2016-08-04 00:01:18 UTC [17370-1] postgres@hooke ERROR: row is too big: size 105344, maximum size 8160
2016-08-04 00:01:18 UTC [17370-2] postgres@hooke CONTEXT: COPY task_000033, line 178992
2016-08-04 00:01:18 UTC [17370-3] postgres@hooke STATEMENT: SELECT worker_merge_files_into_table (1876, 33, '{intermediate_column_1876_0,intermediate_column_1876_1,intermediate_column_1876_2,intermediate_column_1876_3}', '{smallint,smallint,json,integer}')
2016-08-04 00:01:18 UTC [7043-13] WARNING: could not receive query results from localhost:5432
2016-08-04 00:01:18 UTC [7043-14] DETAIL: Client error: row is too big: size 105344, maximum size 8160
This error message is coming from PostgreSQL. However, this [error message](https://www.postgresql.org/message-id/29492.1267795795%40sss.pgh.pa.us) usually exhibits itself when the table declaration has thousands of non-toastable columns.
In this case, we only have four columns. Further, the discrepancy between the row size (105 KB) and maximum size (8 KB) signals that the bug is probably related to repartition joins' table creation logic.
Contributor guide
Research direction
Start with the repartition-join path and the worker_merge_files_into_table call shown in the log, then reproduce the failure with a large JSON column and four-column input. Compare the generated intermediate table definition with the PostgreSQL row-size error. Done means identifying whether repartition joins mishandle JSONB-sized rows and adding coverage or a confirmed fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100