QuantEcon / QuantEcon/workflow-backups

ENH: Optimize GitHub API usage for issues metadata backup

Open
#3 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Summary

The issues metadata backup feature (added in v0.3.0) works but has performance and rate limit concerns that need addressing before enabling in production.

Current Implementation

  • Uses PyGithub REST API
  • 1 API call per page of issues (30 issues/page)
  • 1 API call per issue to fetch comments
  • For a repo with 100 issues → ~100+ API calls

Test Results

Backing up QuantEcon.manual (29 issues):

  • Time: ~38 seconds (~1.3s per issue)
  • API calls: ~30
  • Output: 43 KB JSON file

Concerns for Full Org Backup

  • QuantEcon has ~100 active repos
  • If average 50 issues per repo = 5,000+ API calls
  • GitHub Actions GITHUB_TOKEN limit: 1,000 requests/hour
  • Could easily hit rate limits

Proposed Solutions

Option 1: GraphQL API (Recommended)

Use GitHub GraphQL API to fetch issues + comments in a single query per repo.

query {
  repository(owner: "QuantEcon", name: "quantecon-py") {
    issues(first: 100, states: [OPEN, CLOSED]) {
      nodes {
        number
        title
        body
        comments(first: 100) {
          nodes { author { login } body createdAt }
        }
      }
    }
  }
}

Benefits:

  • Single request per repo (with pagination)
  • Dramatically fewer API calls
  • Faster execution
Option 2: Add include_comments config option
backup_metadata:
  issues: true
  include_comments: false  # Skip comments, much faster
Option 3: Rate limit handling

Add retry logic with exponential backoff when rate limited.

Current Status

  • Feature implemented and tested ✅
  • Default disabled (issues: false) until optimized
  • Config includes warning comment about API usage

Related

  • Issues backup JSON schema is finalized and working
  • Markdown recovery utility planned for future

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing issues metadata backup implementation and its PyGithub calls, then compare the proposed GraphQL, comment-skipping, and retry approaches. Done should mean substantially fewer API requests for repository backups while preserving the finalized issues backup JSON schema and handling GitHub rate limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
api, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.