Do trigger /ingest even if not uploaded anything and if zarr-checksum is different from locally computed
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
NB for now dumping thoughts, no immediate action to take I think.
In https://github.com/dandi/dandi-cli/pull/1047 (fixing https://github.com/dandi/dandi-cli/issues/1046) we stopped calling `/ingest` if we have not uploaded anything to overcome 400 from server saying that there is no need to ingest. Then there was discussion in https://github.com/dandi/dandi-archive/issues/1138#issuecomment-1178053691 suggesting that client might still want/need to trigger `/ingest` process and also having in mind that we cannot revoke upload presigned URLs (https://github.com/dandi/dandi-archive/issues/1162) and thus some "lost" client can still upload some outdated file etc.
My thinking is that if we do have full list of files for zarr prefix on S3 with their checksums/ETags, we do compare to local ones, so we do checksumming, and we can compute/compare `zarr-checksum` known to zarr metadata to the one computed from S3 . If they differ, that means that there were changes to zarr on S3 which were not "caught" and we need to re-digest. So, instead of not calling it, we should call it! But then, most likely, dandi-archive might spit out 400 that no digestion is needed. Then we should issue a warning to contact **us** because I do not think we should allow for unconditional re-ingestion (https://github.com/dandi/dandi-archive/issues/1170).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.