treeverse / treeverse/dvc

Support multiple external cache locations

Open
#4,312 7 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request p3-nice-to-have
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

I have a huge dataset (bigger than my hard drive) but there are several HDDs in my machine. It would be cool if DVC could manage several external caches, and, for example, switch to a new one, when the previous is full (but keep track of the previous one, of course).

If I get it correctly, the core of DVC is a hash table stored on the disk, with very neat features, such as synchronization between different machines.
It seems that combining several such tables is pretty straightforward for all operations except insertion - this is the only place where you need to pick a specific table.
This could be implemented with a wrapper around LocalCache, which takes several instances and a selection strategy and implements the same interface.

Of course I may be oversimplifying things - I am not very familiar with your codebase. Just sharing some thoughts 😉

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating LocalCache and the insertion path it uses. The issue does not name files or tests, so map the existing cache interface before deciding how multiple locations and selection should work. Done would mean supporting several external caches while retaining access to previously used locations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.