bloomreach / bloomreach/s4cmd

Files not always being put to s3

Open
#110 1 comment 0 reactions 0 assignees View on GitHub
info-needed
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

using put from an ec2 to s3, I think we are seeing a hidden error that is not showing up, trying to upload a large number of files, some uploaded but only about half, files are about 5M, no errors being reported from

#!/bin/bash
# s3-copy.sh dest src ...

here=${0%/*}
source $here/job.conf
$scl

dir=$1
shift

for file in "$@"
do
if [ -e "$file" ];
then
base=`basename $file`
s4cmd put --API-ContentEncoding=gzip --API-ContentType=text/tab-separated-values $file s3://$hiveBucket$hiveBucketPrefix/$1/$base
fi
done

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the s4cmd put invocation in s3-copy.sh and review how its output and failures are handled while the loop processes multiple files. Use the supplied script with a large file set and check the resulting S3 contents; done means missing uploads are either prevented or reported clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.