avoid making copy of dataset with "if you have a local copy" functionality
- Vorherrschende Sprache
- Python
- Sterne
- 391
- Forks
- 58
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
**Is your feature request related to a problem? Please describe.**
As suggested by @cmacdonald:
It's annoying that if you already have a copy of a source file and use the "if you have a local copy" functionality, it will make a new copy of it.
**Describe the solution you'd like**
Instead of linking to `~/.ir_datasets/downloads/[hash]`, link directly to where the file will be copied anyway.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
This isn't as straightforward as it may seem on the surface.
- Making this change would mean that verification of the file no longer happens. This is potentially dangerous, as it could mean that people end up using different versions of the dataset, even when they think they are the same. (E.g., if a change was made to the source file or it wasn't downloaded properly or in its entirety.)
- We already make this concession with some non-downloadable datasets. Also, if somebody edits the files in the ~/.ir_datasets directory, there's already this risk.
- We could overcome this to an extent by adding a `.md5_verified` file (or similar) to indicate the file has been checked. This may be the best middle ground. RE: migration, how would we handle files that have already been downloaded?
- Some files undergo a filter between when they are downloaded and when they are saved to the ~/.ir_datasets directory. For instance, files may be unzipped, untarred, etc. These cases would necessarily need to fall back to the current approach. What's the best way to detect if something is further filtered? The presence of a "cache_path" in the download config?
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.