developmentseed / developmentseed/cng-sandbox
feat: multidim ingestion pipeline (skip COG conversion)
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 2h 21m
- Merged PRs (30d)
- 3
Description
## Summary
Create a new ingestion pipeline for multidimensional datasets that skips COG conversion entirely. The raw NetCDF/HDF5 file is stored in R2 and metadata is extracted for use by the titiler-multidim tiler. This is the backend pipeline for #113.
## Context
Currently all NetCDF/HDF5 files go through a conversion pipeline:
1. Scan variables → user picks one → convert to COG → register in pgSTAC → serve via titiler-pgstac
For multidimensional datasets (those with extra dimensions beyond x/y/time, detected by the scanner in the previous issue), we want a different path:
1. Scan variables → detect extra dims → upload raw file to R2 → extract metadata → persist dataset → serve via titiler-multidim
No conversion. No STAC registration. The multidim tiler reads the raw file directly from R2.
## Prerequisites
- Scanner reports `extra_dims` (see scanner enhancement issue)
- `FormatPair.NETCDF_MULTIDIM` / `HDF5_MULTIDIM` exist in the model
## Tasks
### 1. Create `ingestion/src/services/multidim_pipeline.py`
New pipeline module with the following steps:
```python
async def run_multidim_pipeline(
job: Job,
input_path: str,
variables: list[dict], # from scanner, includes extra_dims
emit: Callable, # SSE emitter
) -> None:
```
**Step 1: Upload raw file to R2**
- Reuse existing S3 upload logic from `pipeline.py` / `s3.py`
- Upload to `datasets/{dataset_id}/raw/{filename}`
- Build the S3 URL: `s3://{bucket}/datasets/{id}/raw/{filename}`
**Step 2: Extract spatial metadata**
- Open with xarray: `xr.open_dataset(input_path)`
- Extract bounds (bbox) from spatial coordinates
- Extract CRS if available (via rioxarray or CF conventions)
- Determine min/max zoom from spatial resolution
**Step 3: Collect dimension metadata**
- From the scan results, build `DimensionInfo` list for all extra dims
- Build `multidim_variables` list (names of all eligible data variables)
- For each variable, collect its dimension structure
**Step 4: Build tile URL**
- Template: `{public_multidim_tiler_url}/tiles/{z}/{x}/{y}?url={s3_url}`
- The variable and dimension params get appended by the frontend at render time
- Store just the base URL pattern in the dataset
**Step 5: Persist dataset**
- Create `Dataset` with:
- `is_multidim=True`
- `dimensions=[DimensionInfo(...), ...]`
- `multidim_variables=["var1", "var2", ...]`
- `raw_file_url=s3://{bucket}/...`
- `bounds`, `crs`, `min_zoom`, `max_zoom` from spatial metadata
- `dataset_type="raster"`
- `format_pair="netcdf-multidim"` or `"hdf5-multidim"`
### 2. Wire into main pipeline
**File**: `ingestion/src/services/pipeline.py`
After the scanner runs and before the variable selection flow, add detection:
```python
# Check for extra dimensions
has_extra_dims = any(
v.get("extra_dims") and len(v["extra_dims"]) > 0
for v in variables
)
if has_extra_dims:
# Upgrade format pair
if job.format_pair == FormatPair.NETCDF_TO_COG:
job.format_pair = FormatPair.NETCDF_MULTIDIM
elif job.format_pair == FormatPair.HDF5_TO_COG:
job.format_pair = FormatPair.HDF5_MULTIDIM
await run_multidim_pipeline(job, input_path, variables, emit)
return
```
This intercepts before the `len(variables) > 1` check that triggers the variable picker, so multidim datasets never pause for user selection.
### 3. SSE stage flow
Multidim datasets skip `converting` and `validating` stages. The flow is:
- `scanning` → `ingesting` → `ready`
Emit appropriate status updates:
- During scanning: `detail: "Detected multidimensional dataset with N variables and M dimensions"`
- During ingesting: `detail: "Uploading raw file to storage"`
- On completion: standard `ready` status with dataset_id
The existing frontend handles missing stages gracefully (stages are built from status strings, not a fixed list).
## Verification
- [ ] Upload a simple GeoTIFF → still goes through COG pipeline (backward compat)
- [ ] Upload a simple NetCDF with 1 variable, no time → COG pipeline
- [ ] Upload a NetCDF with 1 variable + time → temporal pipeline
- [ ] Upload a multidimensional NetCDF (extra dims) → multidim pipeline:
- Raw file appears in R2 at `datasets/{id}/raw/{filename}`
- No COG files generated
- Dataset record has `is_multidim: true`
- Dataset has populated `dimensions` and `multidim_variables`
- `raw_file_url` points to the S3 location
- `tile_url` contains the multidim tiler base URL
- [ ] Existing tests pass (`cd ingestion && uv run pytest -v`)
## Files to create/modify
| File | Action |
|------|--------|
| `ingestion/src/services/multidim_pipeline.py` | **Create** |
| `ingestion/src/services/pipeline.py` | Add multidim detection + routing |
Part of #113
Contributor guide
Assessment
This issue has not been assessed yet.