MeltanoLabs / MeltanoLabs/Singer-Working-Group

Reference implementation for `tail=true` setting, enabling indefinitely-running, interruptible processes

Open
#7 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

  • #22 by @pandemicsyn — closed without merging
Applies To::Taps Applies To::Targets Issue Type::Proposal Proposal Type::New Capability
Dominant language
No language data
Stars
15
Forks
4
PR merge metrics
No merged PRs in 30d

Description

For streaming use cases, an always-on tap should be able to "tail" the data source and continually run the extract. This is possible today without changes to the spec, but there is not yet any guidance or "best practice" in terms of how to accomplish this.

A reference implementation would likely entail one or more of the following:

  1. A setting like tail (boolean) which, when true would put the tap into an infinite loop of extract, sleep, extract, and so on.
  2. To honor resumability on interrupt, something like a state_message_min_cadence setting could be established, which forces a state message to flush at least every 1-5 minutes, for instance, assuming 1 or more records have been sent since the last STATE message.
  3. For incremental streams, something like a new_record_polling_interval setting could indicated how much time to sleep between calls to check for new records.
  4. To keep non-incremental streams whole while running a tap in tail mode, we'll want to have a max_full_table_age or similar, which would drive new extracts of streams pulled with FULL_TABLE after the set amount of time has elapsed.
  5. Receiving a SIGTERM message (before SIGKILL) should immediately trigger a flush the latest STATE message. In an ideal scenario, sending the termination command (like ctrl+c) should give the tap a few seconds to send on the last viable state bookmark and give an optimal chance of not having to repeat records on a subsequent execution.

Note: While the priority here is probably the tail setting, it should be fair to assume that a method that needs to run indefinitely should also be ready to be interrupted at any point; since the only way to "stop" the stream is to kill the process, we will need to be ready for interruption and plan for how STATE tracking will be affected by the worst-case scenario interruption timing.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the issue's proposed tail, polling, state-flush, full-table, and SIGTERM behaviors, then inspect linked pull request #22 for prior work. Done would require an agreed reference implementation or guidance covering indefinite execution and interruption-safe state handling; no source files or tests are named.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.