BONSAMURAIS / BONSAMURAIS/vacuum_pump

Design document on where to store data backups / data to be imported

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
good first issue help wanted
Dominant language
Shell
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Most data is available, but we probably don't want to be hitting these external sites too often. At the same point, hosting large files ourselves costs money. Also, it could be the case that we process a lot of raw data and store the results in the DB - what do we do with the raw data, if anything?

Clear guidelines and best practices are needed here. Doesn't need to be perfect the first time around, this isn't the final version!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by reviewing how the project currently obtains external data, imports it, stores processed results in the relational database, and handles raw files. Done means a first-pass design document with clear guidance for backups, external-source access, hosted files, and raw-data retention.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
data-engineering, databases, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.