--mode verify: Two changes to dandisets without update to timestamps
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 5
Description
Followup to prior older
- #49
full log with pointers to more logs
```shell
dandi@drogon:/mnt/backup/dandi/dandisets$ duct flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'
2025-10-07T16:09:43-0400 [INFO ] con-duct: duct is executing 'flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'...
2025-10-07T16:09:43-0400 [INFO ] con-duct: Log files will be written to .duct/logs/2025.10.07T16.09.43-64178_
2025-10-07T16:09:51-0400 [WARNING ] dandi: A newer version (0.71.4) of dandi/dandi-cli is available. You are using 0.66.3
2025-10-07T16:12:30-0400 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
^^^^^^^^^^^^^^^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
changed = await self.sync_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^
dandiset, ds, dmanager, error_on_change=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 251, in sync_dataset
await syncer.prune_deleted()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 93, in prune_deleted
raise UnexpectedChangeError(
...<3 lines>...
)
backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server
2025-10-07T16:29:08-0400 [ERROR ] backups2datalad: Job failed on input :
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
| outp = await func(inp)
| ^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
| changed = await self.sync_dataset(
| ^^^^^^^^^^^^^^^^^^^^^^^^
| dandiset, ds, dmanager, error_on_change=True
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 250, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| ^^^^^^^^^^^^^^^^^^^
| self.dandiset, self.ds, self.manager, self.tracker, self.error_on_change
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/asyncer.py", line 509, in async_assets
| anyio.create_task_group() as nursery,
| ~~~~~~~~~~~~~~~~~~~~~~~^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 815, in __aexit__
| raise BaseExceptionGroup(
| "unhandled errors in a TaskGroup", self._exceptions
| )
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 373, in needs_sync
| local_paths.remove(path)
| ~~~~~~~~~~~~~~~~~~^^^^^^
| KeyError: '.zattrs'
|
| During handling of the above exception, another exception occurred:
|
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 567, in sync_zarr
| await zsync.run()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 152, in run
| if not await self.needs_sync(client, last_sync, to_delete):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 375, in needs_sync
| self.check_change(f"entry {path!r} added")
| ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 497, in check_change
| raise UnexpectedChangeError(
| ...<3 lines>...
| )
| backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server
+------------------------------------
2025-10-07T16:31:45-0400 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 229, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 141, in update_from_backup
raise RuntimeError(
f"Backups for {quantify(len(report.failed), 'Dandiset')} failed"
)
RuntimeError: Backups for 2 Dandisets failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2025/10/2025.10.07.20.09.50Z.log
action summary:
publish (notneeded: 2)
2025-10-07T16:31:45-0400 [INFO ] con-duct: Summary:
Exit Code: 1
Command: flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify
Log files location: .duct/logs/2025.10.07T16.09.43-64178_
Wall Clock Time: 1322.390 sec
Memory Peak Usage (RSS): 1.4 GB
Memory Average Usage (RSS): 581.5 MB
Virtual Memory Peak Usage (VSZ): 8.8 TB
Virtual Memory Average Usage (VSZ): 2.0 TB
Memory Peak Percentage: 1.8%
Memory Average Percentage: 0.6166279969064191%
CPU Peak Usage: 236.4%
Average CPU Usage: 35.82204176334106%
```
so we have two issues
### 1 - deleted file without timestamp update
backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server
### 2 - added to zarr file without timestamp update
backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the `backups2datalad-update-cron --mode verify` entry point and reading `syncer.py` around `prune_deleted`, `asyncer.py` around `async_assets`, and `zarr.py` around `needs_sync`. Investigate how deleted assets and added Zarr entries are handled when the server timestamp is unchanged; done means both reported cases have an agreed, verified outcome without the shown failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100