Make a local cache of extracted raw data for running with extract switched off
- Dominant language
- Python
- Stars
- 5
- Forks
- 3
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 8
Description
Once #282 is complete, CMEW will be able to use previously extracted data and skip extracting data from the MASS archive. For this to work, a user needs to point CMEW to previously extracted data, and this would be easier if we created a central cache for a few model runs, sufficient for the default runs in the standard configuration of CMEW.
Acceptance criteria:
- Given the standard configuration of CMEW, but with extract switched off and `RAW_DATA_DIR` pointing to the new cache
- When CMEW is run
- It works!
Contributor guide
Research direction
Start by examining CMEW's standard configuration, the extraction workflow, and how `RAW_DATA_DIR` is consumed; issue #282 describes the prerequisite behavior. Done means the standard configuration runs successfully with extraction disabled and `RAW_DATA_DIR` pointed at the new cache.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100