Large Distributed Table Upgrades
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Our product uses Citus. When shipping new product versions, we will need to upgrade older schema versions to the latest schema version while preserving data. Citus is targeted for large data pools, and we have a large distributed table that can use well over half the available disk space. Some obvious Postgres table upgrades (UPDATE, ALTER TABLE) require free space equivalent to the table size. So, if one wants to optimize disk usage, and use a large distributed table while maintaining data across upgrades, there is a problem.
Because Citus shards the data into smaller tables "under the hood", it is uniquely positioned to support the "large table upgrade" by upgrading each table separately, and allowing the space to be reclaimed on a per-table basis. Some caveats may include:
- Database is not available for other transactions during the upgrade
- Rollback may be a challenge or not supported.
- Full multi-shard performance may not be available to minimize bloat.
Or, perhaps, there are other approaches (e.g. add another worker node)?
Contributor guide
Assessment
This issue has not been assessed yet.