developmentseed / developmentseed/cng-sandbox
feat: extend scanner to detect extra dimensions in NetCDF/HDF5
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 2h 21m
- Merged PRs (30d)
- 3
Description
## Summary
Extend the ingestion scanner to detect and report dimensions beyond x/y/time in NetCDF and HDF5 files. Add the data model changes needed to represent multidimensional datasets. This is the detection layer for #113.
## Context
The scanner (`ingestion/src/services/scanner.py`) currently identifies:
- **Spatial dims** (x, y, lat, lon, etc.) — used for the raster grid
- **Time dim** — reported as `time_dim` with name, size, and decoded values
But it treats ALL non-time dimensions as spatial. A 5D variable with dims `(time, pest, driver, y, x)` reports `pest` and `driver` as spatial dims and includes them in the shape — which is wrong. These extra dimensions need to be identified separately so the pipeline can route multidim datasets correctly.
## Tasks
### 1. Extend `scan_netcdf()` to report `extra_dims`
**File**: `ingestion/src/services/scanner.py` (line 105-139)
Current logic (line 112):
```python
time_dims = [d for d in da.dims if d.lower() in ("time", "t")]
spatial_dims = [d for d in da.dims if d.lower() not in ("time", "t")]
```
This puts everything that isn't time into `spatial_dims`. Fix: identify the actual spatial dims (last 2 non-time dims, matching coordinate names) and collect the rest as `extra_dims`.
```python
# Identify true spatial dims (the last 2 dims matching known coord names, or just the last 2)
non_time_dims = [d for d in da.dims if d.lower() not in ("time", "t")]
spatial_dims = non_time_dims[-2:] # Last 2 are spatial (y, x)
extra_dims = non_time_dims[:-2] # Everything else is an extra dimension
extra_dim_info = []
for d in extra_dims:
values = None
if d in ds.coords:
try:
values = [str(v) for v in ds.coords[d].values]
except Exception:
pass
extra_dim_info.append({
"name": str(d),
"size": da.sizes[d],
"values": values,
})
```
Add `"extra_dims": extra_dim_info` to each variable dict.
Also fix `spatial_shape` to only use the actual spatial dims (not extra dims):
```python
spatial_shape = [da.sizes[d] for d in spatial_dims]
```
### 2. Extend `scan_hdf5()` to report `extra_dims`
**File**: `ingestion/src/services/scanner.py` (line 56-102)
Currently only handles 2D and 3D datasets (line 77: `if obj.ndim == 3`). For `ndim > 3`, the extra leading dimensions (beyond time and the 2 spatial dims) should be reported as `extra_dims`.
For HDF5, dimension metadata is less structured than NetCDF. Extra dims may not have coordinate arrays. Report them with name `"dim{i}"` and size, with `values: None` unless a matching coordinate dataset is found in the group.
### 3. Add `FormatPair` enum values
**File**: `ingestion/src/models/__init__.py` (line 26-60)
Add to `FormatPair`:
```python
NETCDF_MULTIDIM = "netcdf-multidim"
HDF5_MULTIDIM = "hdf5-multidim"
```
Update the `dataset_type` property to return `DatasetType.RASTER` for these new values.
No change needed in `from_extension` — extension-based detection still returns `NETCDF_TO_COG`/`HDF5_TO_COG`. The format pair gets upgraded after scanning reveals extra dimensions (handled in the pipeline issue).
### 4. Add `DimensionInfo` model and Dataset fields
**File**: `ingestion/src/models/__init__.py`
```python
class DimensionInfo(BaseModel):
name: str
size: int
values: list[str] | None = None
```
Add to the `Dataset` model:
```python
is_multidim: bool = False
dimensions: list[DimensionInfo] = []
multidim_variables: list[str] = []
raw_file_url: str | None = None
```
These serialize into `metadata_json` via the existing persistence layer.
### 5. Add TypeScript types
**File**: `frontend/src/types.ts`
```typescript
export interface DimensionInfo {
name: string;
size: number;
values: string[] | null;
}
```
Add to `Dataset` interface:
```typescript
is_multidim: boolean;
dimensions: DimensionInfo[];
multidim_variables: string[];
raw_file_url: string | null;
```
Add to `MapItem` interface:
```typescript
isMultidim: boolean;
dimensions: DimensionInfo[];
multidimVariables: string[];
rawFileUrl: string | null;
```
Update `datasetToMapItem` in `frontend/src/hooks/useMapData.ts` to map these fields.
## Verification
- [ ] `scan_netcdf()` on a 5D NetCDF (e.g., dims: time, pest, driver, y, x) returns variables with populated `extra_dims`
- [ ] `scan_netcdf()` on a simple 2D or 3D NetCDF returns `extra_dims: []` (backward compat)
- [ ] `scan_hdf5()` on a 4D+ HDF5 file returns variables with populated `extra_dims`
- [ ] `FormatPair.NETCDF_MULTIDIM.dataset_type` returns `DatasetType.RASTER`
- [ ] Existing ingestion tests still pass (`cd ingestion && uv run pytest -v`)
- [ ] Frontend types compile without errors (`cd frontend && npx tsc --noEmit`)
## Files to modify
| File | Change |
|------|--------|
| `ingestion/src/services/scanner.py` | Add extra_dims detection to both scan functions |
| `ingestion/src/models/__init__.py` | Add FormatPair values, DimensionInfo, Dataset fields |
| `frontend/src/types.ts` | Add DimensionInfo, extend Dataset and MapItem |
| `frontend/src/hooks/useMapData.ts` | Map new fields in datasetToMapItem |
Part of #113
Contributor guide
Assessment
This issue has not been assessed yet.