developmentseed / developmentseed/cng-sandbox

feat: create STAC collection from remote storage bucket

Open
#74 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3
Forks
0
Avg merge
2h 21m
Merged PRs (30d)
3

Description

## Summary

Allow users to point the sandbox at a remote storage location (S3 bucket, Azure Blob container, GCS bucket) containing COG files and automatically build a STAC collection from them — without requiring a pre-existing STAC catalog.

This is the complement to the "import existing STAC collection" feature. That feature assumes a catalog already exists; this one builds a catalog from scratch by crawling raw files.

## Motivation

Many users have large collections of imagery sitting in cloud storage with no STAC catalog in front of them. Today, the only way to get that data into the sandbox is to upload files one at a time. This feature would let users connect an entire bucket of COGs and have the sandbox automatically:

1. Discover all COG files in the bucket (with optional prefix/path filter)
2. Extract metadata from each file (bounds, CRS, bands, datetime)
3. Build STAC items and register them in local pgSTAC
4. Serve tiles via titiler-pgstac, with temporal support if datetimes are detected

## Key Design Considerations

### File discovery
- Support S3, Azure Blob, GCS at minimum (all accessible via GDAL virtual filesystem)
- Allow prefix/path filtering (e.g. "only scan `imagery/2024/`")
- Handle large buckets gracefully — paginated listing, progress reporting

### Metadata extraction
- Bounds and CRS from COG headers (GDAL can do this without downloading the full file)
- Datetime extraction from filenames (reuse existing `temporal_ordering.py` logic) and/or COG metadata tags
- Band count and data type for rendering hints (rescale, colormap)

### Handling messy real-world data
- Mixed CRS across files — reproject to common CRS or reject?
- Missing timestamps — fall back to file modification date? alphabetical ordering?
- Mixed resolutions — flag but allow?
- Non-COG GeoTIFFs mixed in — detect and skip (or convert?)

### Authentication
- Same auth challenge as STAC collection import — need credentials to access private buckets
- AWS: access key + secret, or IAM role
- Azure: connection string or SAS token
- GCS: service account key

### Scale
- A bucket could contain thousands or millions of files
- Need async processing with progress reporting (similar to existing upload pipeline SSE)
- Consider pagination and batching for STAC item registration

### Refresh / sync
- Same as STAC import: start with manual refresh (re-crawl bucket, add new items)
- Future: watch for new files via S3 event notifications, etc.

## Relationship to other features

- **Depends on**: STAC collection connection infrastructure (pgSTAC registration, temporal tile serving, connection data model extensions) — being built as part of the STAC collection import feature
- **Shares**: temporal ordering logic, STAC item/collection construction (`stac_ingest.py`), tile URL generation
- **Extends**: the connection model with a new `connection_type` (e.g. `storage_bucket`)

## Out of scope (for initial implementation)

- Auto-sync / file watching
- Converting non-COG files to COG during crawl
- Generating overviews or pre-computing mosaics
- Supporting non-raster files (GeoParquet, etc.) in the bucket

## Open questions

- Should we support mixed file types in a single bucket connection (COGs + non-COGs), or require that the bucket contains only COGs?
- How do we handle very large collections in the UI — virtual scrolling for timesteps, spatial sub-selection?
- Should the crawl happen client-side (browser lists bucket contents) or server-side (ingestion service does the crawl)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.