bloomreach / bloomreach/s4cmd

Better error reporting when MemoryError reading a file

Open
#161 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

s4cmd 2.1.0 outputs an inscrutable error and stack trace when it gets a MemoryError trying to read a file in read_file_chunk().

The debug verbose output does not say anything about the MemoryError, I only see it below because I added a bunch of `print` statements (starting with `!`) and a `try/except` handler in read_file_chunk().

Btw I was running in a container with only 200MB free, trying to write a 1GB file to an S3-compatible system. Hopefully s4cmd is not trying to load the entire file into memory?

```
# s4cmd --debug --verbose --endpoint-url http://backup66:8085 put 1G s3://mybucket/1G
(D)s4cmd.py:662 read S3 keys from $HOME/.s3cfg file
(D)s4cmd.py:128 >> put_handler(, ['put', '1G', 's3://mybucket/1G'])
!!!calling func
(D)s4cmd.py:128 >> validate(, 'cmd|local|s3', ['put', '1G', 's3://mybucket/1G'])
!!!calling func
!!!back from func
(D)s4cmd.py:132 << validate(, 'cmd|local|s3', ['put', '1G', 's3://mybucket/1G']): None
(D)s4cmd.py:128 >> put_files(, ['1G'], 's3://mybucket/1G')
!!!calling func
(D)s4cmd.py:128 >> put_single_file(, , '1G', 's3://mybucket/1G')
!!!calling func
!!!back from func
(D)s4cmd.py:132 << put_single_file(, , '1G', 's3://mybucket/1G'): None
(D)s4cmd.py:128 >> upload(, '1G', 's3://mybucket/1G')
!!!calling func
(D)s4cmd.py:128 >> lookup(, )
!!!calling func
(D)s4cmd.py:404 >> S3APICALL head_object(Bucket='mybucket', Key='1G')
!!!getattr () {'Bucket': 'mybucket', 'Key': '1G'}
!!!back from func
(D)s4cmd.py:132 << lookup(, ): None
(D)s4cmd.py:128 >> read_file_chunk(, '1G', 0, 1073741824)
!!!calling func
!read_file_chunk 1G
!read_file_chunk 1G pos 0
!EXCEPTION: read_file_chunk 1G pos 0:
!!!fail: [Runtime Failure] Unable to read data from source: 1G
(E)s4cmd.py:185 [Runtime Failure] Unable to read data from source: 1G
Exception in thread Thread-1:ad(s)]
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/bin/s4cmd.py", line 530, in run
fail('[Runtime Failure] ', exc_info=e)
File "/usr/bin/s4cmd.py", line 192, in fail
raise RuntimeError(status)
RuntimeError: 1

!!!fail: [Thread Failure] Unable to read data from source: 1G
(E)s4cmd.py:185 [Thread Failure] Unable to read data from source: 1G
!!!fail: [Runtime Exception] 1
(E)s4cmd.py:185 [Runtime Exception] 1
(E)s4cmd.py:187 Traceback (most recent call last):
File "/usr/bin/s4cmd.py", line 1939, in main
CommandHandler(opt).run(args)
File "/usr/bin/s4cmd.py", line 1568, in run
CommandHandler.__dict__[cmd + '_handler'](self, args)
File "/usr/bin/s4cmd.py", line 130, in wrapper
ret = func(*args, **kargs)
File "/usr/bin/s4cmd.py", line 1666, in put_handler
self.s3handler().put_files(source, target)
File "/usr/bin/s4cmd.py", line 130, in wrapper
ret = func(*args, **kargs)
File "/usr/bin/s4cmd.py", line 825, in put_files
pool.join()
File "/usr/bin/s4cmd.py", line 598, in join
self.tasks.join()
File "/usr/bin/s4cmd.py", line 473, in join
fail('[Thread Failure] ', exc_info=self.exc_info)
File "/usr/bin/s4cmd.py", line 192, in fail
raise RuntimeError(status)
RuntimeError: 1
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in s4cmd.py at read_file_chunk(), then follow the thread failure path shown in the traceback through put_files() and ThreadPool.join(). Reproduce the 1GB upload with limited memory and debug output; done means a MemoryError produces a clear, useful failure instead of an inscrutable stack trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.