MeltanoLabs / MeltanoLabs/tap-github
Passing a username as "organizations" config value crashes the tap
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
The following config:
{
"auth_token": access_token,
"start_date": "2022-12-09",
"organizations": ["laurents"]
}
will crash the tap with the following traceback (trimmed to the useful part):
2022-12-09 17:36:11,534 Beginning incremental sync of 'repositories'...
2022-12-09 17:36:11,534 Tap has custom mapper. Using 1 provided map(s).
2022-12-09 17:36:11,900 Tap will run with 1 auth tokens
Traceback (most recent call last):
[...]
File "tap_github/repository_streams.py", line 182, in get_records
yield from super().get_records(context)
File "singer_sdk/streams/rest.py", line 537, in get_records
for record in self.request_records(context):
File "singer_sdk/streams/rest.py", line 357, in request_records
resp = decorated_request(prepared_request, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "singer_sdk/streams/rest.py", line 253, in _request
self.validate_response(response)
File "/tap_github/client.py", line 233, in validate_response
raise FatalAPIError(msg)
singer_sdk.exceptions.FatalAPIError: 404 Client Error: b'{"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#list-organization-repositories"}' (Reason: Not Found) for path: /orgs/laurents/repos
This is because my username is not an org from github's perspective. Calling the users/laurents/repos endpoint works fine on the other hand.
I'm not sure what the best solution is:
- let calling code check that the values are valid. I don't think this is right, the tap shouldn't crash.
- run a preliminary query like we do for repositories and use it to filter out invalid org names.
- handle the error where it's raised, but it's difficult to have enough context to do it properly.
- any other ideas?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the reported organizations configuration, then inspect tap_github/repository_streams.py at get_records and tap_github/client.py at validate_response. Compare the organization and user repository endpoints described in the report. Done should include a regression check showing that a username in organizations receives a defined non-crashing outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100