Support multiple external cache locations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
I have a huge dataset (bigger than my hard drive) but there are several HDDs in my machine. It would be cool if DVC could manage several external caches, and, for example, switch to a new one, when the previous is full (but keep track of the previous one, of course).
If I get it correctly, the core of DVC is a hash table stored on the disk, with very neat features, such as synchronization between different machines.
It seems that combining several such tables is pretty straightforward for all operations except insertion - this is the only place where you need to pick a specific table.
This could be implemented with a wrapper around LocalCache, which takes several instances and a selection strategy and implements the same interface.
Of course I may be oversimplifying things - I am not very familiar with your codebase. Just sharing some thoughts 😉
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating LocalCache and the insertion path it uses. The issue does not name files or tests, so map the existing cache interface before deciding how multiple locations and selection should work. Done would mean supporting several external caches while retaining access to previously used locations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100