citusdata / citusdata/citus

Repartitioning constraint-group task-assignment bug when shard intervals overlap

Open
#458 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

The below diagram captures the scenario occurring here. Because lineitem has overlapping partitions the output of the merge task joins with more than one lineitem shard. We thus create a constraint group involving two SQL and one merge task (ignoring the map-fetch tasks).

In this case we assign the SQL task with lineitem_101 first, and then propagate the assignment to the merge and the other SQL task involving lineitem_102 as well. This fails as lineitem_102 may not be on the same node.

This construct was not built to handle overlapping partitions. As per our initial design document this was because we could have lineitem join with a smaller table (broadcast join) which has more than one shard. We'd thus have two tasks with lineitem_101 in it, and thus need to have both tasks depend on the same merge task.

We have two possible fixes:

**Short term:**
We currently prune data-fetch tasks for the anchor shards for each SQL task. In the below case the anchor shard for both SQL tasks should be lineitem_101, and thus the data-fetch task for lineitem_102 shouldn't be pruned away. This probably isn't a good long-term solution as it involves fetching shards for the big table.

**Long term:**
We currently also prune merge-fetch tasks, which fetch merged data. If we implement these, then that decouples the materialization of the merged data and the use of that data. Each SQL task which depends on that merged data can fetch it independently. This would greatly simplify our task assignment as well as our failure handling logic, and is probably the better long term solution.

![img_20140808_133947-2](https://cloud.githubusercontent.com/assets/2596363/14659383/6a31e00e-0650-11e6-8de1-7db1f01619e6.jpg)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing constraint-group task assignment and the data-fetch and merge-fetch pruning logic described in the issue. Done means overlapping lineitem partitions no longer cause invalid assignment propagation across nodes, with the chosen fix covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.