dlt-hub / dlt-hub/dlt

[core] customize pipeline with hooks

Open
#63 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 15h
Merged PRs (30d)
38

Description

action to be triggered by certain events in the pipeline can be delegated by the pipeline developer. examples of events and actions:
- load job permanently failed. user code gets error message, can inspect the affected file and make decision to proceed or abort the whole load
- new table or new column was inferred. user may change the data type, write disposition or ignore the change (which will drop the data)
- variant column is being created due to inconsistent types. options as above
- value cannot be coerced into the column type. user can provide custom casting function (ie. for text->datetime conversions)
- relational json normalizer recurs into nested table and calls a hook. based on table name, parent table name, nesting level etc. you can decide to create complex type that will serialize the nested elements into json or proceed with recursion or *drop the data*
- when writing text value into the loader file we exceed VARCHAR(MAX) length: user can cut the string or raise error (or ignore and try his luck)

[we need more events]

we need a clean way to define and add hook functions. investigate if https://github.com/getslash/gossip fits.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.