hackforla / hackforla/311-data
DEV - Migrate data pipeline to import GeoJSON into HF
- Dominant language
- JavaScript
- Stars
- 73
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
We should ensure that data flows into our application as GeoJSON, rather than JSON because we require GeoJSON to populate the map. We are also performing an expensive transforming of JSON to GeoJSON that we should remove to improve map performance.
### More Info (optional)
Every time data is rendered onto the map, we do a `forEach` loop over _EVERY_ service request that transforms the data point from JSON into GeoJSON. There is no reason to do this!!! Socrata natively supports downloading data as GeoJSON, we are wasting precious browser memory on this conversion.
### Action Items
- [ ] Modify UpdateHfDataset.py to grab data from Huggingface as GeoJSON, rather than CSV (why are we using CSV, who knows)
- [ ] configure the data upload to save data as GeoParquet (see resources for definitions)
- [ ] Modify the get-data code in our Map component to expect GeoJSON, rather than JSON
- [ ] Remove the code that re-wraps JSON into GeoJSON
### Resources/Instructions
- https://geoparquet.org/
Contributor guide
Assessment
This issue has not been assessed yet.