Use bcdata to find and select which resources to download
- Dominant language
- R
- Stars
- 20
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Previously, the full file paths to the csvs/zips were not specified in the catalogue resources (e.g., [here](https://catalogue.data.gov.bc.ca/dataset/bc-environmental-monitoring-system-results/resource/6aa7f376-a4d3-4fb4-a51c-b4487600d516), so we had to [scrape and parse](https://github.com/bcgov/rems/blob/b1535da182c028a2416bbe4b29e11da0190c5b71/R/get_ems_data.R#L156-L197) this [html table](https://pub.data.gov.bc.ca/datasets/949f2233-9612-4b06-92a9-903e817da659/). We should be able to simplify this greatly using `bcdata::bcdc_get_record()`, `bcdata::bcdc_tidy_resources()` etc.
Contributor guide
Research direction
Start in R/get_ems_data.R, especially lines 156-197, and compare the current HTML-table scraping with the catalogue resource metadata. Read the bcdata functions bcdc_get_record() and bcdc_tidy_resources() first. Done means resource paths can be found and selected through bcdata instead of scraping the HTML table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100