hackforla / hackforla/311-data

DEV - Migrate data pipeline to import GeoJSON into HF

Open
#2,031 1 comment 0 reactions 0 assignees View on GitHub
Complexity: Large on agenda: dev p-feature: data ready for dev lead Role: Frontend size: 3pt
Dominant language
JavaScript
Stars
73
Forks
74
PR merge metrics
No merged PRs in 30d

Description

### Overview

We should ensure that data flows into our application as GeoJSON, rather than JSON because we require GeoJSON to populate the map. We are also performing an expensive transforming of JSON to GeoJSON that we should remove to improve map performance.

### More Info (optional)

Every time data is rendered onto the map, we do a `forEach` loop over _EVERY_ service request that transforms the data point from JSON into GeoJSON. There is no reason to do this!!! Socrata natively supports downloading data as GeoJSON, we are wasting precious browser memory on this conversion.

### Action Items

- [ ] Modify UpdateHfDataset.py to grab data from Huggingface as GeoJSON, rather than CSV (why are we using CSV, who knows)
- [ ] configure the data upload to save data as GeoParquet (see resources for definitions)
- [ ] Modify the get-data code in our Map component to expect GeoJSON, rather than JSON
- [ ] Remove the code that re-wraps JSON into GeoJSON

### Resources/Instructions
- https://geoparquet.org/

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.