protocol of "recovery" from "fatal: a branch named 'release-0.241109.2357' already exists"

NB `001248` is still "recovering" I will close this issue with final notes on it when it is done.

```
2024-11-14T18:03:15-0500 [WARNING ] backups2datalad: Failed [rc=128]: git -c receive.autogc=0 -c gc.auto=0 checkout -b release-0.241109.2357 14c8f938bc97b8075f7c65c5a7416ccec6771753 [cwd=/mnt/backup/dandi/dandisets/001249]

Stdout:

Stderr:

fatal: a branch named 'release-0.241109.2357' already exists

2024-11-14T18:03:15-0500 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 209, in update_dandiset
await self.tag_releases(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 337, in tag_releases
await self.mkrelease(dandiset.for_version(v), ds, push=push, log=log)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 454, in mkrelease
await ds.call_git(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/adataset.py", line 273, in call_git
await aruncmd(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 224, in aruncmd
raise e
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 206, in aruncmd
r = await anyio.run_process(argstrs, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_core/_subprocesses.py", line 85, in run_process
raise CalledProcessError(cast(int, process.returncode), command, output, errors)
subprocess.CalledProcessError: Command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'checkout', '-b', 'release-0.241109.2357', '14c8f938bc97b8075f7c65c5a7416ccec6771753']' returned non-zero exit status 128.
```

first that release is mentioned in
```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandiset-manifests$ grep -l 'release-0.241109.2357' /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.*
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.12.01.02.11Z.log
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.12.02.02.12Z.log
...
```

and it is within that first one it shows that it somehow ended up dirty

```
2024-11-11T20:02:48-0500 [INFO ] backups2datalad: Dandiset 001249: Tagging version 0.241109.2357
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandisets/001248]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandisets/001248]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 status --porcelain --untracked-files=normal --ignore-submodules=none [cwd=/mnt/backup/dandi/dandisets/001174]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Dandiset 001174: Commit made
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Dandiset 001174: Running `git gc`
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 gc [cwd=/mnt/backup/dandi/dandisets/001174]
2024-11-11T20:02:48-0500 [INFO ] backups2datalad: Dandiset 001250: Remote Dandiset has not been modified since last backup; not syncing
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001250]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 log '--grep=\[backups2datalad\]' '--format=%H %aI' [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001250]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: GitHub remote already exists for 001250
2024-11-11T20:02:48-0500 [INFO ] backups2datalad: Dandiset 001250: Tagging releases for Dandiset
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 log '--grep=\[backups2datalad\]' '--format=%H %aI' [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 rev-list '--grep=\[backups2datalad\]' 14c8f938bc97b8075f7c65c5a7416ccec6771753..HEAD [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 rev-list '--grep=\[backups2datalad\]' 14c8f938bc97b8075f7c65c5a7416ccec6771753..HEAD [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show 14c8f938bc97b8075f7c65c5a7416ccec6771753:.dandi/assets.json [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 show 14c8f938bc97b8075f7c65c5a7416ccec6771753:.dandi/assets.json [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:48-0500 [INFO ] backups2datalad: Dandiset 001249: Assets in candidate commits do not match assets in version 0.241109.2357; syncing
2024-11-11T20:02:48-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 checkout -b release-0.241109.2357 14c8f938bc97b8075f7c65c5a7416ccec6771753 [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:49-0500 [INFO ] backups2datalad: Dandiset 001251: Remote Dandiset has not been modified since last backup; not syncing
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001251]
2024-11-11T20:02:49-0500 [INFO ] backups2datalad: Dandiset 001252: Remote Dandiset has not been modified since last backup; not syncing
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001252]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandisets/001250]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001251]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: GitHub remote already exists for 001251
2024-11-11T20:02:49-0500 [INFO ] backups2datalad: Dandiset 001251: Tagging releases for Dandiset
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 remote [cwd=/mnt/backup/dandi/dandisets/001252]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: GitHub remote already exists for 001252
2024-11-11T20:02:49-0500 [INFO ] backups2datalad: Dandiset 001252: Tagging releases for Dandiset
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 show -s --format=%H [cwd=/mnt/backup/dandi/dandisets/001250]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 checkout -b release-0.241109.2357 14c8f938bc97b8075f7c65c5a7416ccec6771753 [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:49-0500 [INFO ] backups2datalad: Dandiset 001249: Syncing
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 status --porcelain --untracked-files=normal --ignore-submodules=none [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:49-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 status --porcelain --untracked-files=normal --ignore-submodules=none [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-11T20:02:49-0500 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 209, in update_dandiset
await self.tag_releases(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 337, in tag_releases
await self.mkrelease(dandiset.for_version(v), ds, push=push, log=log)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 457, in mkrelease
await self.sync_dataset(
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 233, in sync_dataset
raise RuntimeError(f"Dirty {dandiset}; clean or save before running")
RuntimeError: Dirty Dandiset 001249/0.241109.2357; clean or save before running
```

I have reset changes and removed the tag

```shell
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git status
On branch release-0.241109.2357
Changes to be committed:
(use "git restore --staged ..." to unstage)
modified: .dandi/assets-state.json
modified: .dandi/assets.json
modified: dandiset.yaml

(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git reset --hard ; git checkout draft; git brand -D release-0.241109.2357
HEAD is now at 14c8f93 [backups2datalad] 14 files added
Switched to branch 'draft'
Your branch is ahead of 'github/draft' by 2 commits.
(use "git push" to publish your local commits)
git: 'brand' is not a git command. See 'git --help'.

The most similar command is
branch
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git reset --hard ; git checkout draft; git branch -D release-0.241109.2357
HEAD is now at 14c8f93 [backups2datalad] 14 files added
Already on 'draft'
Your branch is ahead of 'github/draft' by 2 commits.
(use "git push" to publish your local commits)
Deleted branch release-0.241109.2357 (was 14c8f93).
```

and rerun sync for that dandiset -- now fails differently: complains about gc'ing a file. digging shows that it was due to running out of space but still committing the rest :-/

```shell
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron 001249
2024-11-14T18:39:27-0500 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 192, in update_dandiset
changed = await self.sync_dataset(dandiset, ds, dmanager)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 246, in sync_dataset
await syncer.dump_asset_metadata()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 111, in dump_asset_metadata
raise UnexpectedChangeError(
backups2datalad.util.UnexpectedChangeError: Dandiset 001249: 1 asset garbage-collected from assets.json: sub-Mount-1/sub-Mount-1_image.nwb
2024-11-14T18:39:27-0500 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 229, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 135, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.14.23.39.25Z.log
```

which relates to this mode of operation

```
README.md: - `gc_assets` *(boolean)* — If set and `assets.json` contains any assets
README.md- neither on the server nor in the backup, delete the extra assets instead
README.md- of erroring. Defaults to `false`.
```

which isn't clear to me how could happen in correct operation. Looking at the commit which added that to `assets.json`, it said that 1 file added but really did not add any file to commit:

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git show --stat 486afa1
commit 486afa1
Author: DANDI User
Date: 2024 Nov 06 00:41:43 -0500

[backups2datalad] 1 file added

.dandi/assets-state.json | 2 +-
.dandi/assets.json | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 152 insertions(+), 2 deletions(-)
```

and that file is mentioned in following logs

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ grep -l sub-Mount-1/sub-Mount-1_image.nwb /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.*
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.06.06.02.12Z.log
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.07.20.22.59Z.log
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.12.00.02.13Z.log
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.14.23.39.25Z.log
```

and looking at the first file we get into the reason

```
2024-11-06T01:03:12-0500 [INFO ] backups2datalad: Dandiset 001249: Finished getting assets from API
2024-11-06T01:03:12-0500 [INFO ] backups2datalad: Dandiset 001249: Asset sub-Mount-1/sub-Mount-1_image.nwb: Syncing
2024-11-06T01:03:12-0500 [INFO ] backups2datalad: Dandiset 001249: Asset sub-Mount-1/sub-Mount-1_image.nwb: Not in dataset; will add
2024-11-06T01:03:12-0500 [DEBUG ] backups2datalad: Running: git -c receive.autogc=0 -c gc.auto=0 rm -f --ignore-unmatch -- sub-Mount-1/sub-Mount-1_image.nwb [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-06T01:03:12-0500 [DEBUG ] backups2datalad: Finished [rc=0]: git -c receive.autogc=0 -c gc.auto=0 rm -f --ignore-unmatch -- sub-Mount-1/sub-Mount-1_image.nwb [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-06T01:03:12-0500 [INFO ] backups2datalad: Dandiset 001249: Asset sub-Mount-1/sub-Mount-1_image.nwb: File is binary; registering key with git-annex
2024-11-06T01:03:12-0500 [DEBUG ] backups2datalad: Opening pipe to `git -c receive.autogc=0 -c gc.auto=0 annex examinekey --batch --migrate-to-backend=SHA256E` [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-06T01:03:12-0500 [DEBUG ] backups2datalad: Opening pipe to `git -c receive.autogc=0 -c gc.auto=0 annex fromkey --force --batch --json --json-error-messages` [cwd=/mnt/backup/dandi/dandisets/001249]
2024-11-06T01:03:12-0500 [ERROR ] backups2datalad: `git annex fromkey SHA256E-s94235823960--2823ac77850c307c2539091e88c4df0f6337669b8d15ea54c820979f9d7d02d7.nwb sub-Mount-1/sub-Mount-1_image.nwb` [cwd=/mnt/backup/dandi/dandisets/001249] call failed: git-annex: /tmp/hash3911726-0: hClose: resource exhausted (No space left on device)
```

which I vaguely remember. and we even got a follow up error

```
RuntimeError: /mnt/backup/dandi/dandisets/001249 is still dirty after committing. Please check if all changes were staged.
```

the last two commits , which includes also the commit claiming to add a file without accomplishing that, are not yet pushed, so I hard reset , and rerun.

```shell
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git reset --hard github/draft
HEAD is now at 923653a [backups2datalad] Only some metadata updates
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron 001249
```

which completes fine and we get release tag as well

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron 001249
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.14.23.54.04Z.log
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 923653a..903309b]
publish(ok): . (dataset) [refs/heads/git-annex->github:refs/heads/git-annex eafd39b..b60965a]
action summary:
publish (ok: 2)
action summary:
publish (notneeded: 2)
Already up to date.
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 248f3b4..b862121]
action summary:
publish (notneeded: 1, ok: 1)
> '[' -s .git/tmp/stderr-2 ']'
> echo 'There was stderr from backup/datalad invocations:'
There was stderr from backup/datalad invocations:
> cat .git/tmp/stderr-2
> backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 5 -e '000(026|108|243|719)$' 001249
> grep -v 'nothing to save, working tree clean'
> git pull --commit --no-edit
> datalad push -J 5
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ git tag
0.241109.2357
```

So, the procedure for the future to handle the cases where we run out of space and detect that

- `git reset --hard`
- if we are on non-draft branch
- `git checkout draft`
- `git delete -D prior-branch`
- review if there were no other tags between `github/draft` and now.
- `git reset --hard github/draft` -- reset to what published before or some other prior reasonable state
- redo backup

Following candidates are to be considered now (removed 1249 from the list)

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001249$ grep ' is still dirty after committing. Please check if all changes were staged' /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.* /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.04.10.02.12Z.log:RuntimeError: /mnt/backup/dandi/dandisets/001169 is still dirty after committing. Please check if all changes were staged.
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.04.16.02.11Z.log:RuntimeError: /mnt/backup/dandi/dandisets/001210 is still dirty after committing. Please check if all changes were staged.
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.04.17.02.11Z.log:RuntimeError: /mnt/backup/dandi/dandisets/001248 is still dirty after committing. Please check if all changes were staged.
/mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.07.20.22.59Z.log:RuntimeError: /mnt/backup/dandi/dandisets/001169 is still dirty after committing. Please check if all changes were staged.

```

### 001169

indeed has commits which were not pushed, which attests to some operations going wrong. no tags or branches. redoing.

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001169$ git reset --hard github/draft; git clean -dfx; /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron $(basename $PWD)
```

### 001210

staged changes, prior commit not pushed... doing the same

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001210$ git reset --hard github/draft; git clean -dfx; /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron $(basename $PWD)
HEAD is now at 729ebd8 [backups2datalad] Only some metadata updates
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.15.00.06.28Z.log
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 729ebd8..a87be5e]
publish(ok): . (dataset) [refs/heads/git-annex->github:refs/heads/git-annex 11c9c8e..08a1a6d]
action summary:
publish (ok: 2)
action summary:
publish (notneeded: 2)
Already up to date.
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft b862121..9670ce4]
action summary:
publish (notneeded: 1, ok: 1)
> '[' -s .git/tmp/stderr-2 ']'
> echo 'There was stderr from backup/datalad invocations:'
There was stderr from backup/datalad invocations:
> cat .git/tmp/stderr-2
> backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 5 -e '000(026|108|243|719)$' 001210
> grep -v 'nothing to save, working tree clean'
> git pull --commit --no-edit
> datalad push -J 5

```

### 001248

seems all good and clear

```shell
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001248$ git lg
* 8830999 - (HEAD -> draft, github/draft) [backups2datalad] 1 file added (7 days ago) [DANDI User]
* 0f9c919 - [backups2datalad] 1 file added (10 days ago) [DANDI User]
...
```

I might have fixed it already -- I remember going through some. Let's verify it though! all good

```shell
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/001248$ /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify $(basename $PWD) Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.15.00.08.55Z.log
Already up to date.
action summary:
publish (notneeded: 2)
> '[' -s .git/tmp/stderr-2 ']'
> echo 'There was stderr from backup/datalad invocations:'
There was stderr from backup/datalad invocations:
> cat .git/tmp/stderr-2
> backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 5 -e '000(026|108|243|719)$' --mode verify 001248
> grep -v 'nothing to save, working tree clean'
> git pull --commit --no-edit
> datalad push -J 5
```

FWIW -- checked that all datasets are in `draft` branch

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets$ grep -v draft */.git/HEAD
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets$
```

### 000953

Found that it had changes uncommitted and not pushed commits, redoing it too

```
(dandisets-2) dandi@drogon:/mnt/backup/dandi/dandisets/000953$ git reset --hard github/draft; git clean -dfx; /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron $(basename $PWD)
HEAD is now at 8dff938 [backups2datalad] Only some metadata updates
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.11.15.00.17.14Z.log
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 8dff938..c6e9895]
publish(ok): . (dataset) [refs/heads/git-annex->github:refs/heads/git-annex 2b25835..8982a87]
action summary:
publish (ok: 2)
Already up to date.
publish(ok): . (dataset) [refs/heads/draft->github:refs/heads/draft 9670ce4..b0f9725]
action summary:
publish (notneeded: 1, ok: 1)
> '[' -s .git/tmp/stderr-2 ']'
> echo 'There was stderr from backup/datalad invocations:'
There was stderr from backup/datalad invocations:
> cat .git/tmp/stderr-2
> grep -v 'nothing to save, working tree clean'
> backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 5 -e '000(026|108|243|719)$' 000953
> git pull --commit --no-edit
> datalad push -J 5

```

### 001246

was found not even fully initialized yet, redoing from scratch while moving original incomplete aside

postponed -- it is the "zarr" one and seems to complain about unclean zarrs