singer-io / singer-io/tap-github
pr_commits table is not ingested properly because of bookmarks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 76
- Forks
- 86
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I just noticed missing data (commits) on the pr_commits table using this integration on Stitch.
The root cause I suspect for this is the incremental mode for this table and the use of bookmarks. Indeed, we fixed the lack of data by triggering a full re-sync of the integration data, but missing commits on this table continued to appear in the future.
Let's take this sample use case :
- Integration bookmark for this table and a given repo is set to :
"pr_commits": {"since": "T"} - Commit A is created at T-2
- Commit B is created a T-1
- Pull Request 1 is opened at T, and includes commits A and B
- Commit C is created at T+1 and associated to PR 1
When ingestion will be triggered again, it will fetch PR 1 and associate it Commit C, as it has been done after T, the current bookmark for the table, but Commits A & B won't be ingested by the integration.
A quick workaround, that won't be cost efficient but will allow to fix the issue is to set this table ingestion mode to full (or allow users to customize this behavior).
It would also be possible to set an offset in days, to allow the retrieval of olders commits in the pr_commits table.
This is a a blocker as it will happen a lot when you have a git workflow using feature branches merged into a develop branch and then picking commits to be merged into main.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the pr_commits incremental ingestion path and how its bookmarks are applied when a pull request gains commits created before the current bookmark. Compare the behavior with full ingestion and verify that commits A and B are not skipped while later commits such as C remain ingested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100