download_PLD OOM-killed reading full geometry for res_id backfill from large global PLD file
- Dominant language
- Python
- Stars
- 3
- Forks
- 1
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
After the fix in #62 gets `download_PLD` past zip extraction, it can still be killed by the OOM killer partway through merging PLD products, e.g.:
```
INFO - found .../SWOT_LakeDatabaseSci_NA_74_..._v202.sqlite (for res_id backfill)
INFO - found .../SWOT_LakeDatabaseSci_NA_75_..._v202.sqlite (for res_id backfill)
INFO - found .../SWOT_LakeDatabase_..._v202_geometries.gpkg (for res_id backfill)
Killed
```
**Root cause:** `_read_pld_file` in `HydroEO/downloaders/hydroweb.py` reads the full geometry column for every tile file used during `res_id` backfill, but the backfill loop only ever uses `lake_id`/`res_id`:
```python
res_id_map.update(dict(zip(tile_gdf["lake_id"], tile_gdf["res_id"])))
```
A PLD download includes a global, non-continent-tagged full-schema file (`..._geometries.gpkg`) alongside the small per-continent `.sqlite` tiles. That global file was observed at **6.0 GB** on disk. Reading it via plain `gpd.read_file(filepath)` with no bounding-box filter and no column subsetting materializes every lake polygon worldwide into memory just to build a `lake_id -> res_id` lookup table, which is enough to exceed available memory (WSL2 in this case) and get the process killed by the OOM killer.
Note: `hydroweb.continent_codes` can be used as a workaround today (restricting backfill to specific continent-tagged tiles excludes the untagged global file), but the underlying inefficiency affects anyone relying on the default "use all full-schema tiles found" behavior.
**Fix:** add a `read_geometry` flag to `_read_pld_file`, and pass `ignore_geometry=True` for backfill-only reads (coverage reads, which do need geometry, are unaffected). Implemented and verified against a live download that previously OOM-killed at this exact step; committed on local branch `fix/pld-backfill-skip-geometry`, not yet opened as a PR.
Co-Authored-By: Claude Sonnet 5
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in HydroEO/downloaders/hydroweb.py at _read_pld_file and follow the res_id backfill loop that reads lake_id and res_id. Compare those reads with coverage reads, which still require geometry, and verify the download no longer materializes the global geometries.gpkg during backfill while coverage behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100