ConduitIO / ConduitIO/conduit

Feature: add record operation `truncate`

Open
#1,654 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

### Feature description

This feature request originated here https://github.com/ConduitIO/conduit-connector-postgres/issues/168.

Some resources (databases) support a truncate operation to delete all records from a collection. In this case, we don't get the full list of records that were deleted, but rather just the information that the collection was truncated. With the current set of operations (`snapshot`, `create`, `update`, `delete`) it's impossible to represent this action. We would have to spawn a `delete` record for every row that was truncated, but the connector doesn't have the information about which rows existed before. Even without this limitation, this approach wouldn't really be performant on big collections.

One solution is to create a `truncate` operation that has to contain the metadata field for which collection is affected. The destination connector could apply this operation as best it can.

We need to figure out what side effects we could experience if we introduce such an operation. For instance, processors working on single records would likely fail to process a `truncate` operation, since the payload and key would likely be empty.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.