developmentseed / developmentseed/cng-sandbox

feat: multidim ingestion pipeline (skip COG conversion)

Open
#116 0 comments 0 reactions 0 assignees View on GitHub
agent-ready feat
Dominant language
TypeScript
Stars
3
Forks
0
Avg merge
2h 21m
Merged PRs (30d)
3

Description

## Summary

Create a new ingestion pipeline for multidimensional datasets that skips COG conversion entirely. The raw NetCDF/HDF5 file is stored in R2 and metadata is extracted for use by the titiler-multidim tiler. This is the backend pipeline for #113.

## Context

Currently all NetCDF/HDF5 files go through a conversion pipeline:
1. Scan variables → user picks one → convert to COG → register in pgSTAC → serve via titiler-pgstac

For multidimensional datasets (those with extra dimensions beyond x/y/time, detected by the scanner in the previous issue), we want a different path:
1. Scan variables → detect extra dims → upload raw file to R2 → extract metadata → persist dataset → serve via titiler-multidim

No conversion. No STAC registration. The multidim tiler reads the raw file directly from R2.

## Prerequisites

- Scanner reports `extra_dims` (see scanner enhancement issue)
- `FormatPair.NETCDF_MULTIDIM` / `HDF5_MULTIDIM` exist in the model

## Tasks

### 1. Create `ingestion/src/services/multidim_pipeline.py`

New pipeline module with the following steps:

```python
async def run_multidim_pipeline(
job: Job,
input_path: str,
variables: list[dict], # from scanner, includes extra_dims
emit: Callable, # SSE emitter
) -> None:
```

**Step 1: Upload raw file to R2**
- Reuse existing S3 upload logic from `pipeline.py` / `s3.py`
- Upload to `datasets/{dataset_id}/raw/{filename}`
- Build the S3 URL: `s3://{bucket}/datasets/{id}/raw/{filename}`

**Step 2: Extract spatial metadata**
- Open with xarray: `xr.open_dataset(input_path)`
- Extract bounds (bbox) from spatial coordinates
- Extract CRS if available (via rioxarray or CF conventions)
- Determine min/max zoom from spatial resolution

**Step 3: Collect dimension metadata**
- From the scan results, build `DimensionInfo` list for all extra dims
- Build `multidim_variables` list (names of all eligible data variables)
- For each variable, collect its dimension structure

**Step 4: Build tile URL**
- Template: `{public_multidim_tiler_url}/tiles/{z}/{x}/{y}?url={s3_url}`
- The variable and dimension params get appended by the frontend at render time
- Store just the base URL pattern in the dataset

**Step 5: Persist dataset**
- Create `Dataset` with:
- `is_multidim=True`
- `dimensions=[DimensionInfo(...), ...]`
- `multidim_variables=["var1", "var2", ...]`
- `raw_file_url=s3://{bucket}/...`
- `bounds`, `crs`, `min_zoom`, `max_zoom` from spatial metadata
- `dataset_type="raster"`
- `format_pair="netcdf-multidim"` or `"hdf5-multidim"`

### 2. Wire into main pipeline

**File**: `ingestion/src/services/pipeline.py`

After the scanner runs and before the variable selection flow, add detection:

```python
# Check for extra dimensions
has_extra_dims = any(
v.get("extra_dims") and len(v["extra_dims"]) > 0
for v in variables
)

if has_extra_dims:
# Upgrade format pair
if job.format_pair == FormatPair.NETCDF_TO_COG:
job.format_pair = FormatPair.NETCDF_MULTIDIM
elif job.format_pair == FormatPair.HDF5_TO_COG:
job.format_pair = FormatPair.HDF5_MULTIDIM

await run_multidim_pipeline(job, input_path, variables, emit)
return
```

This intercepts before the `len(variables) > 1` check that triggers the variable picker, so multidim datasets never pause for user selection.

### 3. SSE stage flow

Multidim datasets skip `converting` and `validating` stages. The flow is:
- `scanning` → `ingesting` → `ready`

Emit appropriate status updates:
- During scanning: `detail: "Detected multidimensional dataset with N variables and M dimensions"`
- During ingesting: `detail: "Uploading raw file to storage"`
- On completion: standard `ready` status with dataset_id

The existing frontend handles missing stages gracefully (stages are built from status strings, not a fixed list).

## Verification

- [ ] Upload a simple GeoTIFF → still goes through COG pipeline (backward compat)
- [ ] Upload a simple NetCDF with 1 variable, no time → COG pipeline
- [ ] Upload a NetCDF with 1 variable + time → temporal pipeline
- [ ] Upload a multidimensional NetCDF (extra dims) → multidim pipeline:
- Raw file appears in R2 at `datasets/{id}/raw/{filename}`
- No COG files generated
- Dataset record has `is_multidim: true`
- Dataset has populated `dimensions` and `multidim_variables`
- `raw_file_url` points to the S3 location
- `tile_url` contains the multidim tiler base URL
- [ ] Existing tests pass (`cd ingestion && uv run pytest -v`)

## Files to create/modify

| File | Action |
|------|--------|
| `ingestion/src/services/multidim_pipeline.py` | **Create** |
| `ingestion/src/services/pipeline.py` | Add multidim detection + routing |

Part of #113

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.