CloudVE / CloudVE/cloudbridge

File integrity check

Open
#76 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
117
Forks
51
Avg merge
9h 17m
Merged PRs (30d)
5

Description

Maybe it would be useful if we could expose means of assessing the integrity of a file. I can think of the following scenarios:

1. check if a file is correctly uploaded to S3. For this, first calculate the file's checksum using a hash function (e.g., MD5, SHA256), then compare it with S3 generated checksum. If the two hash values match, then the file is correctly upload; otherwise retry uploading the file.

2. Get a file's checksum without download the file (e.g., see [this](https://stackoverflow.com/questions/26415923/boto-get-md5-s3-file)). A good scenario for this would checking if a local copy match S3 version, if not, then a collaborator has changed the file, hence update the local copy.

It seems boto is exposing MD5 checksum of an object, however, I guess the challenge would be to (1) ensue if/how other providers expose checksum without downloading a file/object; (2) are all the providers use common hash function for checksum?

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by reviewing the existing cloud provider interfaces and boto's S3 checksum behavior, then determine whether upload verification and remote checksum retrieval can share a cross-provider contract; done means the supported providers and hash semantics are explicitly defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.