citusdata / citusdata/citus

Document and improve range partition support

Open
#1,450 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

##### Feature request
I would like to use the `range` distribution method (see use case below). At the moment, this is possible by manually updating the `pg_dist_partition` and `pg_dist_shard` tables. It would be nice if this functionality would be exposed at a higher level via the `create_distributed_table()` function.

##### Use case
According to the [docs](https://docs.citusdata.com/en/v6.2/performance/scaling_data_ingestion.html#bulk-copy-100-200k-s), the method of choice for bulk ingestion is `COPY`. I have about 200 parallel processes (batch cluster) that all need to insert a lot of data into a distributed table in parallel. The `COPY` approach would probably create too many connections on workers. However, the documentation states that `COPY` will create only connections to workers that hold the shards data will be copied to. So, I would like to explicitly target an individual shard, i.e. "directly" copy data from a compute node to a database worker node without overloading the database servers with too many connections. In order to be able to achieve this, I need to choose values for the partition key such that the records are guaranteed to be co-located on the same shard. This doesn't seem to be possible when these values get hashed.

##### Questions
- Why is the range distribution method not exposed (and documented)?
- Do you see any problems with the above use case? Any suggestions?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.