llnl / llnl/scraper

Error when attempting to access private repo

Open
#30 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
59
Forks
28
PR merge metrics
No merged PRs in 30d

Description

Attempting to run `scraper` on a GitHub org with private repos results in an error.

Output:

```
% scraper --config config.json
2019-04-23 17:29:12,536 - INFO: Connected to: https://github.com
2019-04-23 17:29:12,773 - INFO: Processing: GSA/private-test
Traceback (most recent call last):
File "/home/jf/.pyenv/versions/3.7.0/bin/scraper", line 11, in
load_entry_point('llnl-scraper', 'console_scripts', 'scraper')()
File "/home/jf/gsa/scraper/scraper/gen_code_gov_json.py", line 76, in main
code_json = code_gov.process_config(config_json)
File "/home/jf/gsa/scraper/scraper/code_gov/__init__.py", line 58, in process_config
code_gov_project = Project.from_github3(repo, labor_hours=compute_labor_hours)
File "/home/jf/gsa/scraper/scraper/code_gov/models.py", line 217, in from_github3
elif date_parse(repository.created_at) < POLICY_START_DATE:
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1356, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 645, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 721, in _parse
l = _timelex.split(timestr) # Splits the timestr into tokens
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 207, in split
return list(cls(s))
File "/home/jf/.pyenv/versions/3.7.0/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 76, in __init__
'{itype}'.format(itype=instream.__class__.__name__))
TypeError: Parser must be a string or character stream, not datetime
```

Here is a simplified config.json as a test case. The GSA/private-test repo is private and contains a README.md file.
```
{
"agency": "GSA",
"contact_email": "github-admins@gsa.gov",
"GitHub": [
{
"public_only": false,
"repos": [
"GSA/private-test"
]
}
]
}
```

Example of a real config.json where we encountered the issue. It scans properly until it arrives at a private repo, at which point it crashes.

```
{
"agency": "GSA",
"contact_email": "github-admins@gsa.gov",
"GitHub": [
{
"public_only": false,
"orgs": [
"GSA",
"18F",
"presidential-innovation-fellows",
"USWDS"
],
}
]
}
```

Verified that my GitHub access token is valid and can view private repos by using the same token for a different script.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scraper/code_gov/models.py in Project.from_github3 at the date_parse(repository.created_at) call, then trace how scraper/code_gov/__init__.py process_config supplies repositories from the example config. Run scraper --config config.json against a private repository and make processing complete without the reported TypeError while preserving the public-only behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.