Managed to "conflict on lock" while updating zarr dandisets

not yet sure what/why but during one of the recent updates of 000026 we got into failed state

fatal: Unable to create '/mnt/backup/dandi/dandizarrs/fa9baf63-da85-4623-b84c-551ff510574b/.git/index.lock': File exists.

```shell
fatal: Unable to create '/mnt/backup/dandi/dandizarrs/fa9baf63-da85-4623-b84c-551ff510574b/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
whereis: 17662 failed
2024-04-03T02:42:15-0400 [ERROR ] asyncio: Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'(scanning f...d files...)\n')
handle:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 72, in pipe_data_received
reader.feed_data(data)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 457, in feed_data
assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
git-annex: waitToSetLock: interrupted (Interrupted system call)
whereis: 11749 failed
whereis: 17323 failed
whereis: 7469 failed
whereis: 12217 failed
2024-04-03T02:42:18-0400 [ERROR ] asyncio: Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'ok\n')
handle:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 72, in pipe_data_received
reader.feed_data(data)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 457, in feed_data
assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
2024-04-03T02:42:18-0400 [ERROR ] asyncio: Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'(recording ... in git...)\n')
handle:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 72, in pipe_data_received
reader.feed_data(data)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 457, in feed_data
assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
git-annex: waitToSetLock: interrupted (Interrupted system call)
2024-04-03T02:42:41-0400 [ERROR ] backups2datalad: Job failed on input :
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/aioutil.py", line 177, in dowork
| outp = await func(inp)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 151, in update_dandiset
| changed = await self.sync_dataset(dandiset, ds, dmanager)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 203, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/asyncer.py", line 499, in async_assets
| async with (
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 664, in __aexit__
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/zarr.py", line 574, in sync_zarr
| await ds.push(to="github", jobs=manager.config.jobs, data="nothing")
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/adataset.py", line 297, in push
| await anyio.to_thread.run_sync(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2106, in run_sync_in_worker_thread
| return await future
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 833, in run
| result = context.run(func, *args)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/datalad/distribution/dataset.py", line 507, in apply_func
| return f(*args, **kwargs)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/datalad/interface/base.py", line 773, in eval_func
| return return_func(*args, **kwargs)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/datalad/interface/base.py", line 763, in return_func
| results = list(results)
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/datalad/interface/base.py", line 940, in _execute_command_
| raise IncompleteResultsError(
| datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
| [{'action': 'publish',
| 'hints': None,
| 'message': 'refs/heads/git-annex->github:refs/heads/git-annex [remote '
| 'rejected] (push declined due to repository rule violations)',
| 'operations': ['remote-rejected', 'error'],
| 'path': '/mnt/backup/dandi/dandizarrs/a5908a61-437c-404e-87b6-7023b7c2a169',
| 'refds': '/mnt/backup/dandi/dandizarrs/a5908a61-437c-404e-87b6-7023b7c2a169',
| 'refspec': 'refs/heads/git-annex:refs/heads/git-annex',
| 'status': 'error',
| 'target': 'github',
| 'type': 'dataset'}]
+------------------------------------
2024-04-03T02:42:41-0400 [ERROR ] asyncio: Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'(recording ...d files...)\n')
handle:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/subprocess.py", line 72, in pipe_data_received
reader.feed_data(data)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/streams.py", line 457, in feed_data
assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof
2024-04-03T02:42:41-0400 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/__main__.py", line 228, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/site-packages/backups2datalad/datasetter.py", line 94, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2024.04.03.05.30.14Z.log
Exception ignored in:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 126, in __del__
self.close()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_subprocess.py", line 104, in close
proto.pipe.close()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/unix_events.py", line 546, in close
self._close(None)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/unix_events.py", line 570, in _close
self._loop.call_soon(self._call_connection_lost, exc)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_events.py", line 745, in call_soon
self._check_closed()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.10/asyncio/base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
```

I double checked that that zarr is from 000026

```shell
❯ curl -X 'GET' --silent 'https://api.dandiarchive.org/api/zarr/fa9baf63-da85-4623-b84c-551ff510574b/' -H 'accept: application/json' | jq .
{
"name": "sub-I61/ses-SPIM/micr/sub-I61_ses-SPIM_sample-BrocaAreaS07_stain-Somatostatin_SPIM.ome.zarr",
"dandiset": "000026",
"zarr_id": "fa9baf63-da85-4623-b84c-551ff510574b",
"status": "Complete",
"checksum": "50129984f801bfb4da1a366aeda40820-17746--25629330989",
"file_count": 17746,
"size": 25629330989
}

```

I now reset (`git reset --hard; git clean -dfx`)
```shell
dandi@drogon:/mnt/backup/dandi/dandizarrs/fa9baf63-da85-4623-b84c-551ff510574b$ git reset --hard ; git clean -dfx
Updating files: 100% (10241/10241), done.
HEAD is now at 1a537c8 Exclude .dandi/ from git-annex
Removing 0/
Removing 1/
Removing 2/
Removing 3/
Removing 4/
```

will check recent zarrs (Disabled cron) and will reset 000026 and redo cron...