DOI-USGS / DOI-USGS/dataRetrieval

read_wateruse development idea

Open
#908 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
R
Stars
331
Forks
101
Avg merge
2d 22h
Merged PRs (30d)
6

Description

I took a gander at the new[ `read_wateruse()`](https://github.com/DOI-USGS/dataRetrieval/pull/905) fxn that you and Cee worked on in this MR. It's really nice to have this functionality added to dR, so thank you for those additions.

I wanted to flag one thing for the national/full-CONUS use case, in case it's useful as you think about more development.

Earlier, I was pulling all four irrigation variables for CONUS at HUC12 resolution to make national maps (`wu-irrigation-wd`:` irrwdtot`, `irrwdgw`, `irrwdsw`; `wu-irrigation-cu: irrcutot`).

Because `read_wateruse()` takes one `location` per call and pages at 500 HUC12s, assembling CONUS means looping the 18 HUC2 regions (and for CONUS that is~83,000 HUC12s so roughly 166 paginated requests per variable, I believe). Even after batching all of a model's variables into a single sweep, that's ~330 requests total, and end-to-end it took me over 20 minutes (and I hit an occasional 180s request timeout along the way).

Here's a reproducible example of `read_wateruse()` for CONUS irrigation withdrawals:
```
library(dataRetrieval)
library(purrr)

conus_huc2 <- sprintf("%02d", 1:18) # CONUS HUC2 regions

system.time({
irr_wd_total <- map(conus_huc2, \(h)
read_wateruse(
model = "wu-irrigation-wd",
variable = c("irrwdtot"),
location = paste0("huc2:", h),
timeres = "monthly",
startdate = "2020-01",
enddate = "2020-12"
)
) |> list_rbind()
})
```
For comparison, the approach @aaarcher-usgs and I used previously, pulling the pre-made wide csvs straight from the NWDC file directory, was faster: it's a single file request per variable (the whole CONUS record, already year_month × HUC12), no pagination.

```
base_url <- "https://water.usgs.gov/nwaa-data/data-file-directory?path="
csv_path <- paste0(
"data/water-use/wu-irrigation-wd/irrwdtot/",
"irrwdtot_wu-irrigation-wd_CONUS_200001-202012_wide.csv"
)

system.time({
httr2::request(paste0(base_url, csv_path)) |>
httr2::req_perform(path = "irrwdtot_conus_wide.csv")
})

irr_wd_total <- readr::read_csv("irrwdtot_conus_wide.csv")
```
So for national pulls the file-directory route is one request vs. hundreds. This isn't a blocker at all, `read_wateruse()` is clearlyfully functional works well for subsets of data. But this issue may propose a few ideas for the national-scale case such as:

- Fewer requests for whole-country pulls since the runtime is driven by paginating ~83k HUC12s. Something like a CONUS/"all" location shortcut, an option to hand back the pre-built wide CONUS files for whole-country requests, or a maybe a higher page limit would help (?).
- An option to return wide (`year_month` × `HUC12`) instead of long. This would save the large pivot needed to join to HUC12 geometries for mapping (one column per HUC12). It'd also shrink the payload a bit (long repeats the huc12_id/year_month keys on every row).

These are just a couple of my thoughts after having used it the new function for ongoing national wateruse viz work.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the read_wateruse() entry point and reproduce the documented CONUS HUC2 loop, including its pagination and timing, then compare it with the file-directory CSV route. The issue proposes several alternatives—an all-location shortcut, higher page limits, or wide output—so implementation is not complete until maintainers choose a scope and define performance and output expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.