MeltanoLabs / MeltanoLabs/Singer-Working-Group

Stream-level composability for state artifacts

Open
#6 14 comments 1 reaction 1 assignee View on GitHub

@edgarrmondragon is already working on this.

Since Oct 21, 2021.

Applies To::Taps Issue Type::Proposal Proposal Type::New Capability
Dominant language
No language data
Stars
15
Forks
4
PR merge metrics
No merged PRs in 30d

Description

As discussed in: https://gitlab.com/meltano/meltano/-/issues/2903

Summary: We'd like to be able to run multiple copies of a tap in parallel and then merge the state from the two jobs. While this likely works in practice with most taps today, it requires some assumptions about how state is stored and where/whether merge conflicts are expected.

For example:

An orchestrator kicks off users and customers sync as two separate jobs, running in parallel. If the STATE output from one job contains a top-level users bookmark, and the STATE output from another job contains a top-level customers bookmark, we would like the two job outputs to be able to be merged together to create a single merged STATE output with both users and customers bookmarks combined.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.