Use HDF5
- Dominant language
- JavaScript
- Stars
- 113
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
If you store the data with HDF5, the whole dataset becomes much much smaller. I have 5 months of minute-wise docks available data for citibike stored in ~15Megs. The original json files are 25Gigs. If you put the core data structures into HDF5, you can probably throw them up on s3.
If you remove postgresql, it will be much easier for other people to start hacking on it.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating the current PostgreSQL data path and the JSON dataset handling, then compare an HDF5-based layout and storage workflow. Done would require a defined migration or replacement path, substantially smaller data, and a clear way for others to run the project without PostgreSQL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- data, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100