bloomreach / bloomreach/s4cmd

reconsider use of ETag

Open
#176 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

Given that the ETag algorithm is well known and public at this point, would you reconsider using it? Yes, computing it depends upon knowing the chunk size, but you can use this fact when uploading to confirm that the data is written correctly. You could compute the Etag for any given multipart size while uploading (the md5sum of chunk md5sums) and store the chunk size with the object in custom metadata for retrieval purposes. At the very least, it would provide a very high standard for upload verification and data integrity guarantee.
https://stackoverflow.com/questions/12186993/what-is-the-algorithm-to-compute-the-amazon-s3-etag-for-a-file-larger-than-5gb

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by locating the multipart upload path and existing integrity or metadata handling, then clarify the intended ETag behavior, supported chunk sizes, and verification expectations before determining the relevant test coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.