Comfy-Org / Comfy-Org/download-data

Migrate data fetch + storage to Google Cloud (Cloud Run + GCS)

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
4
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Priority
High — move off GitHub Actions + local SQLite to Google Cloud.

Background
- Fetcher runs in GitHub Actions and writes `data/downloads.db` (SQLite).
- Workflow: `.github/workflows/fetch-data.yml`
- Fetcher: `scripts/fetch-data.mjs`
- API reads SQLite: `app/api/downloads/daily/route.ts`

Goals
- Run data fetching in Google Cloud on a daily schedule.
- Store consumable data in GCS (JSON) so the dashboard can read it without SQLite.
- Keep daily bucketing consistent with `DAILY_CUTOFF_UTC` and current logic.

Proposal (Option A — recommended, minimal code risk)
- Cloud Run Job for the fetcher (containerized):
- Use Secret Manager for `PAT` (GitHub token).
- Trigger daily via Cloud Scheduler at 10:49 UTC (matches `DAILY_CUTOFF_UTC`).
- Fetch with pagination (see issue #3) and apply portable/stable filters (issue #4).
- Emit artifacts to GCS bucket, e.g.:
- `downloads/daily_summary.json`
- `downloads/backfill_events.json`
- (optional) `downloads/asset_daily_stats.jsonl` for debugging.
- Next.js API routes read from GCS JSON instead of SQLite.
- Replace `better-sqlite3` in API with simple fetch/parsing of GCS JSON.
- Leave SQLite usage internal to the job only (or remove it later).

Alternatives
- Option B: Cloud SQL (Postgres) and migrate SQL (replace SQLite-specific syntax).
- Option C: BigQuery for analytics; precompute and cache in GCS for the UI.

Acceptance Criteria
- Daily run on Cloud Run Job updates GCS JSON at/after cutoff time.
- Dashboard loads from GCS-backed API and matches local results (aside from pagination/filters fixes).
- GitHub Actions workflow for fetching is removed or disabled.
- Secrets managed via Secret Manager; no tokens in repo.

Tasks
- [ ] Add Dockerfile + containerize fetcher
- [ ] Implement release pagination (#3)
- [ ] Add portable/stable filters behind env flags (#4)
- [ ] Emit JSON artifacts to GCS (daily_summary, backfill_events)
- [ ] Add Cloud Run Job, Scheduler, and Secret Manager wiring
- [ ] Refactor Next.js API to read from GCS JSON
- [ ] Disable/remove `.github/workflows/fetch-data.yml`
- [ ] Documentation: deployment, env vars, rollback

References
- Fetcher + cutoff: `scripts/fetch-data.mjs` (getBucketDate, DAILY_CUTOFF_UTC)
- Current API: `app/api/downloads/daily/route.ts`
- Current workflow: `.github/workflows/fetch-data.yml`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.