bloomreach / bloomreach/s4cmd

Check md5 after downloading from S3

Open
#90 1 comment 0 reactions 0 assignees View on GitHub
feature-request performance
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

We've recently found a few local files downloaded from S3 using `s4cmd get --sync-check` that were corrupt. Retrying the same download using a separate `s4cmd` invocation resolved the problem (and we have seen the problem on two completely separate, but similarly configured EC2 instances). We were using version 2.0.1.

Since this command already leverages the MD5 hash saved in the S3 metadata (even, apparently, for multi-part S3 objects) it's amazing that the MD5 is not automatically validated against the local copy after the download completes. Although computing the MD5 on even a large local file is fairly quick (given a reasonably powerful system), you could always provide an option to skip such a check in the interest of performance. Ideally, a failed check would be logged and then the download retried (at least `--retry` times).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the implementation of `s4cmd get --sync-check` and the existing `--retry` handling. Review how the command reads S3 metadata and writes local files; done means the downloaded file is checked, failures are logged, and retry behavior follows the requested retry setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.