Files not always being put to s3
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
using put from an ec2 to s3, I think we are seeing a hidden error that is not showing up, trying to upload a large number of files, some uploaded but only about half, files are about 5M, no errors being reported from
#!/bin/bash
# s3-copy.sh dest src ...
here=${0%/*}
source $here/job.conf
$scl
dir=$1
shift
for file in "$@"
do
if [ -e "$file" ];
then
base=`basename $file`
s4cmd put --API-ContentEncoding=gzip --API-ContentType=text/tab-separated-values $file s3://$hiveBucket$hiveBucketPrefix/$1/$base
fi
done
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the s4cmd put invocation in s3-copy.sh and review how its output and failures are handled while the loop processes multiple files. Use the supplied script with a large file set and check the resulting S3 contents; done means missing uploads are either prevented or reported clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100