000108: new errors never seen before

Run finished with some new errors I believe

```
$ time python -m tools.backups2datalad -l WARNING --backup-root /mnt/backup/dandi --config tools/backups2datalad.cfg.yaml update-from-backup --workers 3 000108
...
... some known problems were spotted like https://github.com/dandi/dandisets/issues/298
...
whereis: 351 failed
whereis: 6494 failed
whereis: 2904 failed
fatal: Unable to write new index file
fatal: Unable to write new index file
fatal: Unable to write new index file
fatal: Unable to write new index file
fatal: Unable to write new index file
fatal: Unable to write new index file
2023-01-31T10:56:38-0500 [ERROR ] backups2datalad: Job failed on input :
Traceback (most recent call last):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/aioutil.py", line 168, in dowork
outp = await func(inp)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 145, in update_dandiset
changed = await self.sync_dataset(dandiset, ds, dmanager)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 188, in sync_dataset
await syncer.sync_assets(error_on_change)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/syncer.py", line 36, in sync_assets
self.report = await async_assets(
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/asyncer.py", line 500, in async_assets
nursery.start_soon(dm.read_addurl)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 537, in sync_zarr
await zsync.run()
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 139, in run
if not await self.needs_sync(client, last_sync, local_paths):
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 333, in needs_sync
async for obj in ao:
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/zarr.py", line 407, in aiter_objects
async for page in client.get_paginator("list_objects_v2").paginate(
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/aiobotocore/paginate.py", line 32, in __anext__
response = await self._make_request(current_kwargs)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/aiobotocore/client.py", line 265, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the ListObjectsV2 operation (reached max retries: 4): Please reduce your request rate.
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 513, in
main(_anyio_backend="asyncio")
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1157, in __call__
return anyio.run(self._main, main, args, kwargs, **opts)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1160, in _main
return await main(*args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1076, in main
rv = await self.invoke(ctx)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1687, in invoke
return await _process_result(await sub_ctx.command.invoke(sub_ctx))
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 1434, in invoke
return await ctx.invoke(self.callback, **ctx.params)
File "/home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages/asyncclick/core.py", line 780, in invoke
rv = await rv
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/__main__.py", line 186, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/mnt/backup/dandi/dandisets/tools/backups2datalad/datasetter.py", line 97, in update_from_backup
raise RuntimeError(
RuntimeError: Backups for 1 Dandiset failed

real 1071m35.912s
user 976m43.011s
sys 402m1.266s
```

so we have new AFAIK

- `botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the ListObjectsV2 operation (reached max retries: 4): Please reduce your request rate.`
- `fatal: Unable to write new index file`

I will now check for dirty zarrs, do resets, pop the stashed fix for https://github.com/dandi/dandisets/issues/298, and try again

Re-run the shown backups2datalad command for Dandiset 000108 and inspect the failures in tools/backups2datalad/zarr.py, especially needs_sync() and aiter_objects(), along with the datasetter.py and syncer.py call sites. Compare the S3 SlowDown and Git index errors, then determine whether a successful retry completes the backup without either failure.

Written by the indexing model from the issue text.