MeltanoLabs / MeltanoLabs/Singer-Working-Group
Aggregating current best practices for existing "Standard" features
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 15
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
This came to mind today as I was considering the effect of currently_syncing, this is a standard top-level state key (via singer-python) that allows a tap to resume from the last stream it was syncing upon interruption. Just setting it and skipping streams on resume until you get to the correct stream is fine, but there are some best practices here. E.g., sort streams into an expected order always, reshuffle streams instead of skipping so that all streams will get attempted per successful run, etc.
In what I'm currently thinking of as the Singer "Standard" (a level above the spec where discovery, metadata, catalogs, etc. live), I'm sure there are a lot more semantic usages of standard metadata and/or other state keys that should be understood and documented in a discoverable location.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by cataloging the mentioned Singer standard state and metadata practices, including currently_syncing, and identify a discoverable location for the guidance. Done means the agreed semantic usages and recommended behaviors are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100