graphile / graphile/graphile-engine

[RFC] @graphile/lds2

Open
#662 12 comments 0 reactions 0 assignees View on GitHub
bug enhancement question
Dominant language
JavaScript
Stars
764
Forks
128
PR merge metrics
No merged PRs in 30d

Description

### Feature description

The new package `@graphile/lds2` is in essence a copy of the current implementation with two changes (1 & 2) and one addition (3):

1. Minimal required version of [wal2json](https://github.com/eulerto/wal2json) is at this time of writing 2.3, due to requiring the `include-pk` parameter to be available. This is required for `@graphile/subscriptions-lds` to properly work in cases where `REPLICA IDENTITY FULL` is enabled on a table.
2. Dropping support for `version-format 1`, and moving to `version-format 2` which no longer reports changes on a transaction level, but on a tuple level. Additional bonus is that in my opinion the json object is structured more like what the industry standard is today.
3. Adding watch capability much to what I believe `graphile-build` has for building it schema. Currently `@graphile/lds` is not able to actively listen to schema changes, adding this would allow us to circumvent the need for the `include-pk` parameter and as such also keep supporting `version-format 1` and [wal2json](https://github.com/eulerto/wal2json) <= 2.3, yet this is not the main goal of this addition.

I believe the most useful purpose of adding a watch capability is to be able to support subscriptions on native pg >=11 partitioned tables and [timescale hypertables](https://docs.timescale.com/latest/faq#hypertable-chunks). These table structures are providing a `virtual` table that one can do `insert`,`update`,`delete`,`select` upon, but in actuality these operations are redirected to one or more sub tables (partitions). [wal2json](https://github.com/eulerto/wal2json) does NOT report changes on this (`virtual`) parent table level, but on the actual subtable that the change occurred. This makes it impossible to subscribe to these type of tables with the current implementation of `@graphile/lds`. I am aware of the fact that native pg >= 11 partitioned tables are NOT very well supported in `postgraphile` atm [(Only the partitions are showing up in the schema, not the virtual table)](https://github.com/graphile/postgraphile/issues/1160), but due to the different technical implementation of [timescale hypertables](https://docs.timescale.com/latest/faq#hypertable-chunks) these are actually coming out in the generated schema just fine (No partitions are shown, just the parent table).

Having this watch capability allows us to figure out that when a change occurs in one of these sub tables, we also need to fabricate a same change for the (`virtual`) parent table.

### Motivating example

Motivation for this alternative version is twofold, being that the the current implementation of `@graphile/lds`:

1. Assumes that the primary key information of a change is always included in the property `oldkeys` if the property `oldkeys` is included. This is NOT always the case due to the possibilty of [enabling `REPLICA IDENTITY FULL` on a table](https://github.com/eulerto/wal2json/blob/a5d764fa1ef9d8c9c67a8cd381dad1381495c92d/wal2json.c#L2318) which changes the contents of `oldkeys` from not only containing the primary key(s), but to containing the entire record as it was before said change (much like the `OLD` value in update/delete triggers). As such, enabling this feature on a table **breaks** the current implementation of `@graphile/lds`.

Fixing the above issue requires the use of a parameter `include-pk` which is only available in version >=2.3 of [wal2json](https://github.com/eulerto/wal2json), and as of right now we haven't been able to find a method to detect which version of [wal2json](https://github.com/eulerto/wal2json) is installed. The only way seems to be that of `try {} catch {}` the code with the parameter and falling back on one without if it fails with a parameter unknown error. Problem I have with this approach is that we'd have to resort to string matching to determine the cause, which feels a bit too iffy to me.

2. Is built on `version-format 1` of [wal2json](https://github.com/eulerto/wal2json) which produces a JSON object per transaction. All of the new/old tuples are available in the JSON object. Whereas the newer `version-format 2` produces a JSON object per tuple, supports the `truncate` operation and provides in my opinion a more logical constructed object.

Below are both formats as defined in `typescript` definitions, these are both the default format which can be augmented with additional metadata based on parameters. One of which is the much needed `include-pk` which is only available in [wal2json](https://github.com/eulerto/wal2json) >= 2.3

`format-version 1`:
```typescript
export interface Change {
kind: Kind;
schema: string;
table: string;
columnnames?: string[];
columntypes?: string[];
columnvalues?: Columnvalue[];
oldkeys?: Oldkeys;
}

enum Kind {
Insert = 'insert',
Update = 'update',
Delete = 'delete'
}

export type Columnvalue = any;

export interface Oldkeys {
keynames: string[];
keytypes: string[];
keyvalues: Keyvalue[];
}

export type Keyvalue = string;
```

`format-version 2`:
```typescript
export interface Change {
action: Action;
schema: string;
table: string;
columns?: Column[];
identity?: Column[];
}

enum Action {
Insert = 'I',
Update = 'U',
Delete = 'D',
Truncate = 'T'
}

export interface Column {
name: string;
type: string;
value: Value;
}

export type Value = any;
```

As we discussed in Discord fixing/adding the above 2 points would result in such breaking changes that are whole new package is warranted. This new package `@graphile/lds2` would have more strict requirements on which version of [wal2json](https://github.com/eulerto/wal2json) is used.

### Breaking changes

None, because it is a new package. Yet, in order to make this package useful an update to `@graphile/subscription-lds` or alternative package that does the same but using `@graphile/lds2` is required.

### Supporting development

I [tick all that apply]:

- [x] am interested in building this feature myself
- [x] am interested in collaborating on building this feature
- [x] am willing to help testing this feature before it's released
- [x] am willing to write a test-driven test suite for this feature (before it exists)
- [ ] am a [Graphile sponsor](https://www.graphile.org/sponsor/) ❤️
- [ ] have an active [support or consultancy contract](https://www.graphile.org/support/) with Graphile

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.