Error when attempting to access private repo
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 59
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Description
Attempting to run `scraper` on a GitHub org with private repos results in an error.
Output:
```
% scraper --config config.json
2019-04-23 17:29:12,536 - INFO: Connected to: https://github.com
2019-04-23 17:29:12,773 - INFO: Processing: GSA/private-test
Traceback (most recent call last):
File "/home/jf/.pyenv/versions/3.7.0/bin/scraper", line 11, in
load_entry_point('llnl-scraper', 'console_scripts', 'scraper')()
File "/home/jf/gsa/scraper/scraper/gen_code_gov_json.py", line 76, in main
code_json = code_gov.process_config(config_json)
File "/home/jf/gsa/scraper/scraper/code_gov/__init__.py", line 58, in process_config
code_gov_project = Project.from_github3(repo, labor_hours=compute_labor_hours)
File "/home/jf/gsa/scraper/scraper/code_gov/models.py", line 217, in from_github3
elif date_parse(repository.created_at) < POLICY_START_DATE:
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1356, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 645, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 721, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 76, in __init__
'{itype}'.format(itype=instream.__class__.__name__))
TypeError: Parser must be a string or character stream, not datetime
```
Here is a simplified config.json as a test case. The GSA/private-test repo is private and contains a README.md file.
```
{
"agency": "GSA",
"contact_email": "github-admins@gsa.gov",
"GitHub": [
{
"public_only": false,
"repos": [
"GSA/private-test"
]
}
]
}
```
Example of a real config.json where we encountered the issue. It scans properly until it arrives at a private repo, at which point it crashes.
```
{
"agency": "GSA",
"contact_email": "github-admins@gsa.gov",
"GitHub": [
{
"public_only": false,
"orgs": [
"GSA",
"18F",
"presidential-innovation-fellows",
"USWDS"
],
}
]
}
```
Verified that my GitHub access token is valid and can view private repos by using the same token for a different script.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scraper/code_gov/models.py in Project.from_github3 at the date_parse(repository.created_at) call, then trace how scraper/code_gov/__init__.py process_config supplies repositories from the example config. Run scraper --config config.json against a private repository and make processing complete without the reported TypeError while preserving the public-only behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100