developmentseed / developmentseed/cng-sandbox

feat: extend scanner to detect extra dimensions in NetCDF/HDF5

Open
#115 0 comments 0 reactions 0 assignees View on GitHub
agent-ready feat
Dominant language
TypeScript
Stars
3
Forks
0
Avg merge
2h 21m
Merged PRs (30d)
3

Description

## Summary

Extend the ingestion scanner to detect and report dimensions beyond x/y/time in NetCDF and HDF5 files. Add the data model changes needed to represent multidimensional datasets. This is the detection layer for #113.

## Context

The scanner (`ingestion/src/services/scanner.py`) currently identifies:
- **Spatial dims** (x, y, lat, lon, etc.) — used for the raster grid
- **Time dim** — reported as `time_dim` with name, size, and decoded values

But it treats ALL non-time dimensions as spatial. A 5D variable with dims `(time, pest, driver, y, x)` reports `pest` and `driver` as spatial dims and includes them in the shape — which is wrong. These extra dimensions need to be identified separately so the pipeline can route multidim datasets correctly.

## Tasks

### 1. Extend `scan_netcdf()` to report `extra_dims`

**File**: `ingestion/src/services/scanner.py` (line 105-139)

Current logic (line 112):
```python
time_dims = [d for d in da.dims if d.lower() in ("time", "t")]
spatial_dims = [d for d in da.dims if d.lower() not in ("time", "t")]
```

This puts everything that isn't time into `spatial_dims`. Fix: identify the actual spatial dims (last 2 non-time dims, matching coordinate names) and collect the rest as `extra_dims`.

```python
# Identify true spatial dims (the last 2 dims matching known coord names, or just the last 2)
non_time_dims = [d for d in da.dims if d.lower() not in ("time", "t")]
spatial_dims = non_time_dims[-2:] # Last 2 are spatial (y, x)
extra_dims = non_time_dims[:-2] # Everything else is an extra dimension

extra_dim_info = []
for d in extra_dims:
values = None
if d in ds.coords:
try:
values = [str(v) for v in ds.coords[d].values]
except Exception:
pass
extra_dim_info.append({
"name": str(d),
"size": da.sizes[d],
"values": values,
})
```

Add `"extra_dims": extra_dim_info` to each variable dict.

Also fix `spatial_shape` to only use the actual spatial dims (not extra dims):
```python
spatial_shape = [da.sizes[d] for d in spatial_dims]
```

### 2. Extend `scan_hdf5()` to report `extra_dims`

**File**: `ingestion/src/services/scanner.py` (line 56-102)

Currently only handles 2D and 3D datasets (line 77: `if obj.ndim == 3`). For `ndim > 3`, the extra leading dimensions (beyond time and the 2 spatial dims) should be reported as `extra_dims`.

For HDF5, dimension metadata is less structured than NetCDF. Extra dims may not have coordinate arrays. Report them with name `"dim{i}"` and size, with `values: None` unless a matching coordinate dataset is found in the group.

### 3. Add `FormatPair` enum values

**File**: `ingestion/src/models/__init__.py` (line 26-60)

Add to `FormatPair`:
```python
NETCDF_MULTIDIM = "netcdf-multidim"
HDF5_MULTIDIM = "hdf5-multidim"
```

Update the `dataset_type` property to return `DatasetType.RASTER` for these new values.

No change needed in `from_extension` — extension-based detection still returns `NETCDF_TO_COG`/`HDF5_TO_COG`. The format pair gets upgraded after scanning reveals extra dimensions (handled in the pipeline issue).

### 4. Add `DimensionInfo` model and Dataset fields

**File**: `ingestion/src/models/__init__.py`

```python
class DimensionInfo(BaseModel):
name: str
size: int
values: list[str] | None = None
```

Add to the `Dataset` model:
```python
is_multidim: bool = False
dimensions: list[DimensionInfo] = []
multidim_variables: list[str] = []
raw_file_url: str | None = None
```

These serialize into `metadata_json` via the existing persistence layer.

### 5. Add TypeScript types

**File**: `frontend/src/types.ts`

```typescript
export interface DimensionInfo {
name: string;
size: number;
values: string[] | null;
}
```

Add to `Dataset` interface:
```typescript
is_multidim: boolean;
dimensions: DimensionInfo[];
multidim_variables: string[];
raw_file_url: string | null;
```

Add to `MapItem` interface:
```typescript
isMultidim: boolean;
dimensions: DimensionInfo[];
multidimVariables: string[];
rawFileUrl: string | null;
```

Update `datasetToMapItem` in `frontend/src/hooks/useMapData.ts` to map these fields.

## Verification

- [ ] `scan_netcdf()` on a 5D NetCDF (e.g., dims: time, pest, driver, y, x) returns variables with populated `extra_dims`
- [ ] `scan_netcdf()` on a simple 2D or 3D NetCDF returns `extra_dims: []` (backward compat)
- [ ] `scan_hdf5()` on a 4D+ HDF5 file returns variables with populated `extra_dims`
- [ ] `FormatPair.NETCDF_MULTIDIM.dataset_type` returns `DatasetType.RASTER`
- [ ] Existing ingestion tests still pass (`cd ingestion && uv run pytest -v`)
- [ ] Frontend types compile without errors (`cd frontend && npx tsc --noEmit`)

## Files to modify

| File | Change |
|------|--------|
| `ingestion/src/services/scanner.py` | Add extra_dims detection to both scan functions |
| `ingestion/src/models/__init__.py` | Add FormatPair values, DimensionInfo, Dataset fields |
| `frontend/src/types.ts` | Add DimensionInfo, extend Dataset and MapItem |
| `frontend/src/hooks/useMapData.ts` | Map new fields in datasetToMapItem |

Part of #113

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.