dssg / dssg/bikeshare

Use HDF5

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
113
Forks
31
PR merge metrics
No merged PRs in 30d

Description

If you store the data with HDF5, the whole dataset becomes much much smaller. I have 5 months of minute-wise docks available data for citibike stored in ~15Megs. The original json files are 25Gigs. If you put the core data structures into HDF5, you can probably throw them up on s3.

If you remove postgresql, it will be much easier for other people to start hacking on it.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the current PostgreSQL data path and the JSON dataset handling, then compare an HDF5-based layout and storage workflow. Done would require a defined migration or replacement path, substantially smaller data, and a clear way for others to run the project without PostgreSQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
data, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.