singer-io / singer-io/tap-github

pr_commits table is not ingested properly because of bookmarks

Open
#185 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
76
Forks
86
PR merge metrics
No merged PRs in 30d

Description

Hello,
I just noticed missing data (commits) on the pr_commits table using this integration on Stitch.

The root cause I suspect for this is the incremental mode for this table and the use of bookmarks. Indeed, we fixed the lack of data by triggering a full re-sync of the integration data, but missing commits on this table continued to appear in the future.

Let's take this sample use case :

  • Integration bookmark for this table and a given repo is set to : "pr_commits": {"since": "T"}
  • Commit A is created at T-2
  • Commit B is created a T-1
  • Pull Request 1 is opened at T, and includes commits A and B
  • Commit C is created at T+1 and associated to PR 1

When ingestion will be triggered again, it will fetch PR 1 and associate it Commit C, as it has been done after T, the current bookmark for the table, but Commits A & B won't be ingested by the integration.

A quick workaround, that won't be cost efficient but will allow to fix the issue is to set this table ingestion mode to full (or allow users to customize this behavior).
It would also be possible to set an offset in days, to allow the retrieval of olders commits in the pr_commits table.

This is a a blocker as it will happen a lot when you have a git workflow using feature branches merged into a develop branch and then picking commits to be merged into main.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the pr_commits incremental ingestion path and how its bookmarks are applied when a pull request gains commits created before the current bookmark. Compare the behavior with full ingestion and verify that commits A and B are not skipped while later commits such as C remain ingested.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.