hackforla / hackforla/311-data
DEV - Create Huggingface repositories to sort annual data into quarters
- Dominant language
- JavaScript
- Stars
- 73
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
### Dependency
- [ ] We need to see that there is a demand for Huggingface as a backup to the Socrata API before we proceed
- every time we add sub-categorizations of data, we need to write logic to "bridge" the datasets
- this adds complexity that I think we should avoid unless we find that Socrata API is just not reliable
### Overview
We need to create quarterly Huggingface Repositories so that we can load smaller chunks of data when using Hugginface, which will increase load times and lead to a better user experience.
### More Information
We will start with quarters for 2024, simply because each repository will require extra work to ensure our cron job is updating each repo correctly. Completing this work can open up the same work to be done on 2023 and years prior. While this ticket is not blocked by the ticket to create Hf repo for 2025 (https://github.com/hackforla/311-data/issues/1895), it is closely related.
### Action Items
- [ ] PM or dev lead: create the following Hugginface Repositories
- [ ] 2024Q1
- [ ] 2024Q2
- [ ] 2024Q3
- [x] [2024Q4](https://huggingface.co/datasets/311-data/2024Q4)
- [ ] modify `hfClean()` to additionally create quarterly datasets locally. Consider using naming convention similar to the ones we make in Huggingface
- [ ] modify `hfUpload()` to also upload those files to the quarterly data repos
### Resources/Instructions
Related Tickets
- https://github.com/hackforla/311-data/issues/1895
Other Resources
- https://huggingface.co/311-data
- https://duckdb.org/docs/
- https://duckdb.org/docs/stable/sql/functions/datepart
- https://duckdb.org/docs/stable/sql/functions/datepart#quarterdate
Contributor guide
Assessment
This issue has not been assessed yet.