developmentseed / developmentseed/cng-sandbox

source.coop: periodic re-sync for growing products

Open
#201 0 comments 0 reactions 0 assignees View on GitHub
enhancement feat
Dominant language
TypeScript
Stars
3
Forks
0
Avg merge
2h 21m
Merged PRs (30d)
3

Description

Part of the source.coop integration roadmap. Depends on #197 (shared example datasets) and ideally #198 (full GHRSST).

## Why

Example datasets register as a snapshot at startup. But some source.coop products grow over time — GHRSST adds a new daily file every day. Without re-sync, the sandbox's view of the catalog stays frozen at the point of first deploy.

Since example datasets are shared (not per-workspace), re-sync runs once globally, not per user. Cheap, high-leverage.

## What

### Scheduled task
- Add a periodic task to the ingestion service (e.g., via APScheduler or a simple async loop) that re-runs each example product's enumerator on a cadence (daily is sufficient for GHRSST; weekly for less-active products).
- For each product:
- Enumerate current items from source.coop
- Query existing pgSTAC items in the product's collection
- Diff: new items → insert, removed items → delete (rare but possible if a publisher rewrites history)
- Unchanged items → no-op

### Configuration
- Add a \`resync_interval\` field to the \`SourceCoopProduct\` dataclass so each product can specify its cadence (default 24h). Non-growing products (GEBCO, Land & Carbon Lab) can set a long interval or skip re-sync entirely.

### Observability
- Log each re-sync run: product slug, items before, items added, items removed, duration.
- Surface the last re-sync timestamp somewhere (e.g., a \`GET /api/datasets/{id}\` field) so the frontend can display "Last updated: 2 hours ago" on example dataset cards.

## Acceptance

- [ ] Re-sync runs on a schedule and logs its results
- [ ] Manually triggering a re-sync (e.g., by deleting an item from pgSTAC) results in the item being re-inserted on the next run
- [ ] Configurable per-product cadence works
- [ ] No impact on user-facing request latency — re-sync runs in the background

## Notes

- The diff logic should be idempotent and safe to interrupt (e.g., if the service is restarted mid-resync).
- Worth measuring GHRSST's full-enumeration time after #198 lands. If it's minutes, re-sync cadence should account for that (don't schedule so tightly that a new run starts before the previous one finishes).
- Consider whether to expose a manual \`POST /api/admin/resync\` endpoint for debugging. Optional.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.