avoid making copy of dataset with "if you have a local copy" functionality
- Linguagem predominante
- Python
- Estrelas
- 391
- Forks
- 58
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
**Is your feature request related to a problem? Please describe.**
As suggested by @cmacdonald:
It's annoying that if you already have a copy of a source file and use the "if you have a local copy" functionality, it will make a new copy of it.
**Describe the solution you'd like**
Instead of linking to `~/.ir_datasets/downloads/[hash]`, link directly to where the file will be copied anyway.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
This isn't as straightforward as it may seem on the surface.
- Making this change would mean that verification of the file no longer happens. This is potentially dangerous, as it could mean that people end up using different versions of the dataset, even when they think they are the same. (E.g., if a change was made to the source file or it wasn't downloaded properly or in its entirety.)
- We already make this concession with some non-downloadable datasets. Also, if somebody edits the files in the ~/.ir_datasets directory, there's already this risk.
- We could overcome this to an extent by adding a `.md5_verified` file (or similar) to indicate the file has been checked. This may be the best middle ground. RE: migration, how would we handle files that have already been downloaded?
- Some files undergo a filter between when they are downloaded and when they are saved to the ~/.ir_datasets directory. For instance, files may be unzipped, untarred, etc. These cases would necessarily need to fall back to the current approach. What's the best way to detect if something is further filtered? The presence of a "cache_path" in the download config?
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.