singer-io / singer-io/tap-github

Re-add `files` and `stats` to `PrCommits` stream

Open
#202 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
76
Forks
86
PR merge metrics
No merged PRs in 30d

Description

While testing out this tap, I noticed that, similarly to what happens in other variants of the tap-github out there, the Commits stream does not have the files and stats properties, as these ones can only be fetched commit by commit , but not when listing commits. These properties were once available in the Commits stream schema, but it became confusing for users that selected the stream to not see these data points materialised in their target (creating issues like this), hence they were removed by @dsprayberry in this PR.

In this same PR, the files and stats properties were also removed from the PrCommits stream (as they are also commit related streams). Although, unlike the Commits stream, these properties are available while listing commits for a PR, hence could be available in the PrCommits stream, even if they're not available in the Commits stream.

Functionally speaking, I'd also prefer that Github would keep their Commit related endpoints aligned and make files and stats available in both. Unfortunately this is not the case, but I still see the analytical value of having these fields available, even if only for commits that are linked to a Pull Request.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the PrCommits stream and the GitHub “list commits on a pull request” endpoint linked in the issue. Confirm how commit fields are mapped and how the stream schema is defined; done means files and stats are available and materialized for PrCommits records, without changing the Commits stream.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
api, data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.