MeltanoLabs / MeltanoLabs/Singer-Working-Group
Stream-level composability for state artifacts
@edgarrmondragon is already working on this.
Since Oct 21, 2021.
- Dominant language
- No language data
- Stars
- 15
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
As discussed in: https://gitlab.com/meltano/meltano/-/issues/2903
Summary: We'd like to be able to run multiple copies of a tap in parallel and then merge the state from the two jobs. While this likely works in practice with most taps today, it requires some assumptions about how state is stored and where/whether merge conflicts are expected.
For example:
An orchestrator kicks off
usersandcustomerssync as two separate jobs, running in parallel. If the STATE output from one job contains a top-levelusersbookmark, and the STATE output from another job contains a top-levelcustomersbookmark, we would like the two job outputs to be able to be merged together to create a single merged STATE output with bothusersandcustomersbookmarks combined.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.