dandi / dandi/backups2datalad

--mode verify: Two changes to dandisets without update to timestamps

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
1
Avg merge
2d 16h
Merged PRs (30d)
5

Description

Followup to prior older
- #49

full log with pointers to more logs

```shell
dandi@drogon:/mnt/backup/dandi/dandisets$ duct flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'
2025-10-07T16:09:43-0400 [INFO ] con-duct: duct is executing 'flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'...
2025-10-07T16:09:43-0400 [INFO ] con-duct: Log files will be written to .duct/logs/2025.10.07T16.09.43-64178_

2025-10-07T16:09:51-0400 [WARNING ] dandi: A newer version (0.71.4) of dandi/dandi-cli is available. You are using 0.66.3
2025-10-07T16:12:30-0400 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
^^^^^^^^^^^^^^^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
changed = await self.sync_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^
dandiset, ds, dmanager, error_on_change=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 251, in sync_dataset
await syncer.prune_deleted()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 93, in prune_deleted
raise UnexpectedChangeError(
...<3 lines>...
)
backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server
2025-10-07T16:29:08-0400 [ERROR ] backups2datalad: Job failed on input :
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
| outp = await func(inp)
| ^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
| changed = await self.sync_dataset(
| ^^^^^^^^^^^^^^^^^^^^^^^^
| dandiset, ds, dmanager, error_on_change=True
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 250, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| ^^^^^^^^^^^^^^^^^^^
| self.dandiset, self.ds, self.manager, self.tracker, self.error_on_change
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/asyncer.py", line 509, in async_assets
| anyio.create_task_group() as nursery,
| ~~~~~~~~~~~~~~~~~~~~~~~^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 815, in __aexit__
| raise BaseExceptionGroup(
| "unhandled errors in a TaskGroup", self._exceptions
| )
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 373, in needs_sync
| local_paths.remove(path)
| ~~~~~~~~~~~~~~~~~~^^^^^^
| KeyError: '.zattrs'
|
| During handling of the above exception, another exception occurred:
|
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 567, in sync_zarr
| await zsync.run()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 152, in run
| if not await self.needs_sync(client, last_sync, to_delete):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 375, in needs_sync
| self.check_change(f"entry {path!r} added")
| ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 497, in check_change
| raise UnexpectedChangeError(
| ...<3 lines>...
| )
| backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server
+------------------------------------
2025-10-07T16:31:45-0400 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 229, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 141, in update_from_backup
raise RuntimeError(
f"Backups for {quantify(len(report.failed), 'Dandiset')} failed"
)
RuntimeError: Backups for 2 Dandisets failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2025/10/2025.10.07.20.09.50Z.log
action summary:
publish (notneeded: 2)
2025-10-07T16:31:45-0400 [INFO ] con-duct: Summary:
Exit Code: 1
Command: flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify
Log files location: .duct/logs/2025.10.07T16.09.43-64178_
Wall Clock Time: 1322.390 sec
Memory Peak Usage (RSS): 1.4 GB
Memory Average Usage (RSS): 581.5 MB
Virtual Memory Peak Usage (VSZ): 8.8 TB
Virtual Memory Average Usage (VSZ): 2.0 TB
Memory Peak Percentage: 1.8%
Memory Average Percentage: 0.6166279969064191%
CPU Peak Usage: 236.4%
Average CPU Usage: 35.82204176334106%

```

so we have two issues

### 1 - deleted file without timestamp update

backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server

### 2 - added to zarr file without timestamp update

backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the `backups2datalad-update-cron --mode verify` entry point and reading `syncer.py` around `prune_deleted`, `asyncer.py` around `async_assets`, and `zarr.py` around `needs_sync`. Investigate how deleted assets and added Zarr entries are handled when the server timestamp is unchanged; done means both reported cases have an agreed, verified outcome without the shown failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.