citusdata / citusdata/citus

Large Distributed Table Upgrades

Open
#2,500 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Our product uses Citus. When shipping new product versions, we will need to upgrade older schema versions to the latest schema version while preserving data. Citus is targeted for large data pools, and we have a large distributed table that can use well over half the available disk space. Some obvious Postgres table upgrades (UPDATE, ALTER TABLE) require free space equivalent to the table size. So, if one wants to optimize disk usage, and use a large distributed table while maintaining data across upgrades, there is a problem.

Because Citus shards the data into smaller tables "under the hood", it is uniquely positioned to support the "large table upgrade" by upgrading each table separately, and allowing the space to be reclaimed on a per-table basis. Some caveats may include:
- Database is not available for other transactions during the upgrade
- Rollback may be a challenge or not supported.
- Full multi-shard performance may not be available to minimize bloat.

Or, perhaps, there are other approaches (e.g. add another worker node)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.