citusdata / citusdata/pg_shard

Support having multiple head nodes

Open
#37 1 comment 0 reactions 0 assignees View on GitHub
feature research
Dominant language
C
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

The missing feature for pg_shard-as-OLTP-big-data tool is the ability to have multiple head nodes so that you can support thousands of connections. This shouldn't be too hard to solve, and may actually be solvable with existing PostgreSQL 9.4 tools.
1. replicate all of the metadata tables only: these don't get updated that often, so [BDR](https://wiki.postgresql.org/wiki/BDR_Project) ought to be adequate for it.
2. streaming replication option. This one is hampered by (a) needing to figure out which node is the master, and (b) pg_shards' use of temporary tables.
3. FDW option: have metadata be an FDW back to the master node. Likely to add latency and complexity. Also fails to provide us with easy failover from the master node.

Contributor guide

Open the contributing guide

Research direction

This is a design proposal for multiple head nodes in pg_shard, with no files or tests identified. Start by comparing the three mentioned approaches—BDR metadata replication, streaming replication, and an FDW to the master—against master discovery, failover, and temporary-table use. Done would require a selected approach and defined behavior for those constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.