MeltanoLabs / MeltanoLabs/tap-github
Stream `extra_metrics` fails on repos with large number of issues/PRs
Open
@laurentS is already working on this.
Since May 12, 2023.
bug
- Dominant language
- Python
- Stars
- 20
- Forks
- 60
- Avg merge
- 20h 29m
- Merged PRs (30d)
- 6
Description
When running the tap on https://github.com/microsoft/TypeScript with the extra_metrics stream, it crashes because the number of open issues is shown as 5k+ on the project page.
When navigating to https://github.com/microsoft/TypeScript/issues the actual number is 5988 (as of writing this), so really closer to 6k.
Stack trace:
File "tap_github/repository_streams.py", line 2071, in parse_response
yield from scrape_metrics(response, self.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "tap_github/scraping.py", line 126, in scrape_metrics
issues = parse_counter(soup.find("span", id="issues-repo-tab-count"), logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "tap_github/scraping.py", line 109, in parse_counter
return int(title_string.strip().replace(",", ""))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '5000+'
It would make sense to source the number of open issues and PRs from the graphql api endpoint instead.
I will open a PR to fix this.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.