Allow to backup and restore pipeline working directory
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
Background
Backup and restore functions allow to work with snapshots of the pipeline working directory. Working directory contains full pipeline state and load packages and having such functions allows for example:
- take a snapshot after extract step and use it to load the same extracted data into several destinations (we can call it "forking" pipeline)
- take a snapshot after normalize step and use it to load data to several destinations of the same type (ie. two postgres databases) or different type (if file format is compatible: you can load same parquet files to duckdb, bigquery and snowflake)
- take snapshot after load/run to save pipeline state (and schemas) for destinations that do not support state sync (ie. sinks)
4.. take snapshot after failed load to finalize it later - take snapshots after extract or normalize and keep them to be able to "replay" loading
- use snapshot to pass packages between independently working extract, normalize and load steps ie. to scale things horizontally.
Note: snapshot must be restored in local file system, dlt does not support fsspec to keep its working folder (but it works on fuse)
Requirements
Implementation idea is to have helper functions that take pipeline instance, filesystem destination and snapshot name as an input, zip the pipeline state and load packages and pass it to a bucket.
Restore works in opposite way
Would be cool to have a context manager in which we can wrap pipeline run that would make sure that state is backed up and restored on run (sink support)
Would be cool to somehow chunk zipped files if too large
User should be able to tell helper function what to backup (only state, also data, also completed packages)
We should support pipeline fork on restore by allowing to change the pipeline name and destination
PoC: https://gist.github.com/rudolfix/ee6e16d8671f26ac4b9ffc915ad24b6e
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked PoC and compare it with the requirements for snapshot contents, local restoration, pipeline forks, and filesystem destinations. Define the backup and restore API and its supported options before implementation; done means the agreed snapshot cases can be saved and restored without losing the pipeline state or packages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100