Document and improve range partition support
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
##### Feature request
I would like to use the `range` distribution method (see use case below). At the moment, this is possible by manually updating the `pg_dist_partition` and `pg_dist_shard` tables. It would be nice if this functionality would be exposed at a higher level via the `create_distributed_table()` function.
##### Use case
According to the [docs](https://docs.citusdata.com/en/v6.2/performance/scaling_data_ingestion.html#bulk-copy-100-200k-s), the method of choice for bulk ingestion is `COPY`. I have about 200 parallel processes (batch cluster) that all need to insert a lot of data into a distributed table in parallel. The `COPY` approach would probably create too many connections on workers. However, the documentation states that `COPY` will create only connections to workers that hold the shards data will be copied to. So, I would like to explicitly target an individual shard, i.e. "directly" copy data from a compute node to a database worker node without overloading the database servers with too many connections. In order to be able to achieve this, I need to choose values for the partition key such that the records are guaranteed to be co-located on the same shard. This doesn't seem to be possible when these values get hashed.
##### Questions
- Why is the range distribution method not exposed (and documented)?
- Do you see any problems with the above use case? Any suggestions?
Contributor guide
Assessment
This issue has not been assessed yet.