[core] customize pipeline with hooks
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
action to be triggered by certain events in the pipeline can be delegated by the pipeline developer. examples of events and actions:
- load job permanently failed. user code gets error message, can inspect the affected file and make decision to proceed or abort the whole load
- new table or new column was inferred. user may change the data type, write disposition or ignore the change (which will drop the data)
- variant column is being created due to inconsistent types. options as above
- value cannot be coerced into the column type. user can provide custom casting function (ie. for text->datetime conversions)
- relational json normalizer recurs into nested table and calls a hook. based on table name, parent table name, nesting level etc. you can decide to create complex type that will serialize the nested elements into json or proceed with recursion or *drop the data*
- when writing text value into the loader file we exceed VARCHAR(MAX) length: user can cut the string or raise error (or ignore and try his luck)
[we need more events]
we need a clean way to define and add hook functions. investigate if https://github.com/getslash/gossip fits.
Contributor guide
Assessment
This issue has not been assessed yet.