creativecommons / creativecommons/quantifying

Swallowing Critical GitHub API Errors (KeyError) in github_fetch.py

Open
#300 0 comments 1 reaction 0 assignees View on GitHub
🏁 status: ready for work 💻 aspect: code 🛠 goal: fix help wanted 🟩 priority: low
Dominant language
Python
Stars
48
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Description
In `scripts/1-fetch/github_fetch.py`, the script fetches public repository counts for various Creative Commons tools. When parsing the GitHub API search response, the code expects to find `"total_count"` in the returned JSON object:
`count = search_data["total_count"]`

However, if the GitHub API returns an error response (such as "API Rate Limit Exceeded" or "Invalid Token" / "Bad credentials"), `"total_count"` will not exist in the response payload. As a result, the script encounters a `KeyError` on `"total_count"`, which is caught by:
```python
except KeyError as e:
raise shared.QuantifyingException(f"KeyError: {e}", 1)
```
This swallows the actual error details returned by the GitHub API (e.g., in the `"message"` field), making debugging extremely difficult without manually outputting the API response payload.

## Reproduction
1. Run `python scripts/1-fetch/github_fetch.py` using a rate-limited network or an invalid/expired `GH_TOKEN`.
2. The GitHub API returns a JSON response lacking the `"total_count"` key (e.g., containing `"message": "API rate limit exceeded..."` or `"message": "Bad credentials"`).
3. The script throws `KeyError: 'total_count'`, which is converted into `QuantifyingException("KeyError: 'total_count'", 1)`.
4. Observe the script exiting with a generic `KeyError: 'total_count'` exception, concealing the actual API rate limiting or authentication message.

## Expectation
The script should check for GitHub API error messages (like `"message"` or other error indicators in the JSON response) before attempting to access `"total_count"`, logging the specific error payload from GitHub to provide clarity on why the request failed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.