MeltanoLabs / MeltanoLabs/Singer-Working-Group
Reference implementation for `tail=true` setting, enabling indefinitely-running, interruptible processes
Nobody has claimed this yet.
- #22 by @pandemicsyn — closed without merging
- Dominant language
- No language data
- Stars
- 15
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
For streaming use cases, an always-on tap should be able to "tail" the data source and continually run the extract. This is possible today without changes to the spec, but there is not yet any guidance or "best practice" in terms of how to accomplish this.
A reference implementation would likely entail one or more of the following:
- A setting like
tail(boolean) which, whentruewould put the tap into an infinite loop of extract, sleep, extract, and so on. - To honor resumability on interrupt, something like a
state_message_min_cadencesetting could be established, which forces a state message to flush at least every 1-5 minutes, for instance, assuming 1 or more records have been sent since the last STATE message. - For incremental streams, something like a
new_record_polling_intervalsetting could indicated how much time to sleep between calls to check for new records. - To keep non-incremental streams whole while running a tap in
tailmode, we'll want to have amax_full_table_ageor similar, which would drive new extracts of streams pulled withFULL_TABLEafter the set amount of time has elapsed. - Receiving a
SIGTERMmessage (beforeSIGKILL) should immediately trigger a flush the latest STATE message. In an ideal scenario, sending the termination command (likectrl+c) should give the tap a few seconds to send on the last viable state bookmark and give an optimal chance of not having to repeat records on a subsequent execution.
Note: While the priority here is probably the tail setting, it should be fair to assume that a method that needs to run indefinitely should also be ready to be interrupted at any point; since the only way to "stop" the stream is to kill the process, we will need to be ready for interruption and plan for how STATE tracking will be affected by the worst-case scenario interruption timing.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the issue's proposed tail, polling, state-flush, full-table, and SIGTERM behaviors, then inspect linked pull request #22 for prior work. Done would require an agreed reference implementation or guidance covering indefinite execution and interruption-safe state handling; no source files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100