hackforla / hackforla/311-data

DEV - Create Huggingface repositories to sort annual data into quarters

Open
#1,923 3 comments 0 reactions 0 assignees View on GitHub
Complexity: Medium Dependency p-feature: data Role: Data Science size: 3pt
Dominant language
JavaScript
Stars
73
Forks
74
PR merge metrics
No merged PRs in 30d

Description

### Dependency
- [ ] We need to see that there is a demand for Huggingface as a backup to the Socrata API before we proceed
- every time we add sub-categorizations of data, we need to write logic to "bridge" the datasets
- this adds complexity that I think we should avoid unless we find that Socrata API is just not reliable

### Overview

We need to create quarterly Huggingface Repositories so that we can load smaller chunks of data when using Hugginface, which will increase load times and lead to a better user experience.

### More Information

We will start with quarters for 2024, simply because each repository will require extra work to ensure our cron job is updating each repo correctly. Completing this work can open up the same work to be done on 2023 and years prior. While this ticket is not blocked by the ticket to create Hf repo for 2025 (https://github.com/hackforla/311-data/issues/1895), it is closely related.

### Action Items

- [ ] PM or dev lead: create the following Hugginface Repositories
- [ ] 2024Q1
- [ ] 2024Q2
- [ ] 2024Q3
- [x] [2024Q4](https://huggingface.co/datasets/311-data/2024Q4)
- [ ] modify `hfClean()` to additionally create quarterly datasets locally. Consider using naming convention similar to the ones we make in Huggingface
- [ ] modify `hfUpload()` to also upload those files to the quarterly data repos

### Resources/Instructions

Related Tickets
- https://github.com/hackforla/311-data/issues/1895

Other Resources
- https://huggingface.co/311-data
- https://duckdb.org/docs/
- https://duckdb.org/docs/stable/sql/functions/datepart
- https://duckdb.org/docs/stable/sql/functions/datepart#quarterdate

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.