dandi / dandi/dandisets

save changes to dandiset.yaml last

Open
#143 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
15
Forks
2
PR merge metrics
No merged PRs in 30d

Description

ATM, in case of crashes, like whenever some sha256 is not known for an asset, we seems to end up in a dirty dataset state

```
dandi@drogon:/mnt/backup/dandi/dandisets$ git status
On branch draft
Your branch is up to date with 'github/draft'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
(commit or discard the untracked or modified content in submodules)

modified: 000026 (modified content)

no changes added to commit (use "git add" and/or "git commit -a")
dandi@drogon:/mnt/backup/dandi/dandisets$ cd 000026/
dandi@drogon:/mnt/backup/dandi/dandisets/000026$ git status
On branch draft
Your branch is up to date with 'github/draft'.

Changes to be committed:
(use "git reset HEAD ..." to unstage)

modified: dandiset.yaml
```

since it seems changes to dandiset.yaml are saved first. Would it be easy to save them last, right before calling datalad's save on the dandiset? Then there would be a better chance to remain in a clean state for subsequent cron job to possibly succeed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the workflow in this repository that writes each dandiset.yaml and then calls DataLad's save, starting with the code responsible for processing asset sha256 values. Trace the ordering around a crash caused by an unknown sha256. Done means dandiset.yaml is saved only immediately before the dandiset save, reducing the chance that a failed run leaves the dataset dirty.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.