G-Node / G-Node/gin-cli

File metadata write is slow

Open
#269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14
Forks
7
PR merge metrics
No merged PRs in 30d

Description

The metadata filename feature was added in 604176d4642599d496deab6bbc1fbf9b8528524b (Apr 2018).

Feature description:
After annex add is finished, the client writes the filename to the content metadata so that the annex content key is always associated with the name of the file it was added as. This is useful for knowing which file is being transferred when uploading old versions of files (when multiple commits happen between uploads).

Problem:
The metadata writing can take a while when a large number of files is being added, so printing this information is useful.

Solutions:
- Find another way to keep track of key->filename mappings.
- Don't write metadata and simply print the key (hash) of a file when it's being uploaded (like git-annex does).
- Write metadata concurrently during a commit. This was the case for a while but could lead to race conditions (should investigate more).

Additional info:
The current solution doesn't take into account that keys can be mapped to multiple file names. Filenames that have the same content will share the same content key (same hash) and currently this isn't handled in any way. Instead, the last filename to be added simply overwrites any existing value. If we support multiple filenames, we would have to have a way of choosing which one to display during a `git annex copy`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.