MeltanoLabs / MeltanoLabs/tap-github

Passing a username as "organizations" config value crashes the tap

Open
#173 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
20
Forks
60
Avg merge
20h 29m
Merged PRs (30d)
6

Description

The following config:

{
    "auth_token": access_token,
    "start_date": "2022-12-09",
    "organizations": ["laurents"]
}

will crash the tap with the following traceback (trimmed to the useful part):

2022-12-09 17:36:11,534 Beginning incremental sync of 'repositories'...
2022-12-09 17:36:11,534 Tap has custom mapper. Using 1 provided map(s).
2022-12-09 17:36:11,900 Tap will run with 1 auth tokens
Traceback (most recent call last):
[...]
  File "tap_github/repository_streams.py", line 182, in get_records
    yield from super().get_records(context)
  File "singer_sdk/streams/rest.py", line 537, in get_records
    for record in self.request_records(context):
  File "singer_sdk/streams/rest.py", line 357, in request_records
    resp = decorated_request(prepared_request, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "singer_sdk/streams/rest.py", line 253, in _request
    self.validate_response(response)
  File "/tap_github/client.py", line 233, in validate_response
    raise FatalAPIError(msg)
singer_sdk.exceptions.FatalAPIError: 404 Client Error: b'{"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#list-organization-repositories"}' (Reason: Not Found) for path: /orgs/laurents/repos

This is because my username is not an org from github's perspective. Calling the users/laurents/repos endpoint works fine on the other hand.

I'm not sure what the best solution is:

  • let calling code check that the values are valid. I don't think this is right, the tap shouldn't crash.
  • run a preliminary query like we do for repositories and use it to filter out invalid org names.
  • handle the error where it's raised, but it's difficult to have enough context to do it properly.
  • any other ideas?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the reported organizations configuration, then inspect tap_github/repository_streams.py at get_records and tap_github/client.py at validate_response. Compare the organization and user repository endpoints described in the report. Done should include a regression check showing that a username in organizations receives a defined non-crashing outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.