dandi / dandi/dandi-cli

Do trigger /ingest even if not uploaded anything and if zarr-checksum is different from locally computed

Open
#1,049 0 comments 0 reactions 0 assignees View on GitHub
cmd-upload zarr
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

NB for now dumping thoughts, no immediate action to take I think.

In https://github.com/dandi/dandi-cli/pull/1047 (fixing https://github.com/dandi/dandi-cli/issues/1046) we stopped calling `/ingest` if we have not uploaded anything to overcome 400 from server saying that there is no need to ingest. Then there was discussion in https://github.com/dandi/dandi-archive/issues/1138#issuecomment-1178053691 suggesting that client might still want/need to trigger `/ingest` process and also having in mind that we cannot revoke upload presigned URLs (https://github.com/dandi/dandi-archive/issues/1162) and thus some "lost" client can still upload some outdated file etc.

My thinking is that if we do have full list of files for zarr prefix on S3 with their checksums/ETags, we do compare to local ones, so we do checksumming, and we can compute/compare `zarr-checksum` known to zarr metadata to the one computed from S3 . If they differ, that means that there were changes to zarr on S3 which were not "caught" and we need to re-digest. So, instead of not calling it, we should call it! But then, most likely, dandi-archive might spit out 400 that no digestion is needed. Then we should issue a warning to contact **us** because I do not think we should allow for unconditional re-ingestion (https://github.com/dandi/dandi-archive/issues/1170).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.