DHI / DHI/HydroEO

download_PLD OOM-killed reading full geometry for res_id backfill from large global PLD file

Open
#63 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
1
Avg merge
8m
Merged PRs (30d)
1

Description

After the fix in #62 gets `download_PLD` past zip extraction, it can still be killed by the OOM killer partway through merging PLD products, e.g.:

```
INFO - found .../SWOT_LakeDatabaseSci_NA_74_..._v202.sqlite (for res_id backfill)
INFO - found .../SWOT_LakeDatabaseSci_NA_75_..._v202.sqlite (for res_id backfill)
INFO - found .../SWOT_LakeDatabase_..._v202_geometries.gpkg (for res_id backfill)
Killed
```

**Root cause:** `_read_pld_file` in `HydroEO/downloaders/hydroweb.py` reads the full geometry column for every tile file used during `res_id` backfill, but the backfill loop only ever uses `lake_id`/`res_id`:

```python
res_id_map.update(dict(zip(tile_gdf["lake_id"], tile_gdf["res_id"])))
```

A PLD download includes a global, non-continent-tagged full-schema file (`..._geometries.gpkg`) alongside the small per-continent `.sqlite` tiles. That global file was observed at **6.0 GB** on disk. Reading it via plain `gpd.read_file(filepath)` with no bounding-box filter and no column subsetting materializes every lake polygon worldwide into memory just to build a `lake_id -> res_id` lookup table, which is enough to exceed available memory (WSL2 in this case) and get the process killed by the OOM killer.

Note: `hydroweb.continent_codes` can be used as a workaround today (restricting backfill to specific continent-tagged tiles excludes the untagged global file), but the underlying inefficiency affects anyone relying on the default "use all full-schema tiles found" behavior.

**Fix:** add a `read_geometry` flag to `_read_pld_file`, and pass `ignore_geometry=True` for backfill-only reads (coverage reads, which do need geometry, are unaffected). Implemented and verified against a live download that previously OOM-killed at this exact step; committed on local branch `fix/pld-backfill-skip-geometry`, not yet opened as a PR.

Co-Authored-By: Claude Sonnet 5

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in HydroEO/downloaders/hydroweb.py at _read_pld_file and follow the res_id backfill loop that reads lake_id and res_id. Compare those reads with coverage reads, which still require geometry, and verify the download no longer materializes the global geometries.gpkg during backfill while coverage behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.