filecoin-project / filecoin-project/lily

Lily should support migrations for Timescale cluster nodes

Open
#836 0 comments 0 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Go
Stars
52
Forks
47
PR merge metrics
No merged PRs in 30d

Description

### Description
Currently, Lily migrations expect to be deployed to a single Timescale instance. We are currently exploring using lily data in a cluster environment. The cluster environment requires different migrations from their single node case. While I think Lily's migration must support multiple database targets (vanilla postgres + timescale cluster, in addition to the single node setup), the scope of this story is specific to supporting timescale cluster migrations only.

(It's worth noting that I think adding a separate branch of support for vanilla postgres might benefit by whatever pattern gets used here, even if we don't pursue vanilla postgres support today.)

### Acceptance criteria
- I can pass a bool arg or adjust a config value (or control in some other way via CLI) for `lily migrate` to enable the special cluster migration case without affecting the single node deployment case (which should be the default behavior when the new bool is not specified).

### Where to begin

The schema migrations primarily change in the two following ways:
- `create_hypertable` becomes `create_distributed_hypertable` with one extra argument which describes the columnar indicies. (the argument is a (set of?) strings)
- `create type * as enum` and `create function` both require the same statement (as-is) to be executed via `call distributed_exec` which distributes the command to the cluster.

An example of the full schema creation SQL (including the schema creation, which is not currently a concern of lily migrations) used in the cluster environment can be found at https://gist.github.com/placer14/c044e7e8c3aa4933a847ed3c29b59c46.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.