Better error reporting when MemoryError reading a file
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
s4cmd 2.1.0 outputs an inscrutable error and stack trace when it gets a MemoryError trying to read a file in read_file_chunk().
The debug verbose output does not say anything about the MemoryError, I only see it below because I added a bunch of `print` statements (starting with `!`) and a `try/except` handler in read_file_chunk().
Btw I was running in a container with only 200MB free, trying to write a 1GB file to an S3-compatible system. Hopefully s4cmd is not trying to load the entire file into memory?
```
# s4cmd --debug --verbose --endpoint-url http://backup66:8085 put 1G s3://mybucket/1G
(D)s4cmd.py:662 read S3 keys from $HOME/.s3cfg file
(D)s4cmd.py:128 >> put_handler(, ['put', '1G', 's3://mybucket/1G'])
!!!calling func
(D)s4cmd.py:128 >> validate(, 'cmd|local|s3', ['put', '1G', 's3://mybucket/1G'])
!!!calling func
!!!back from func
(D)s4cmd.py:132 << validate(, 'cmd|local|s3', ['put', '1G', 's3://mybucket/1G']): None
(D)s4cmd.py:128 >> put_files(, ['1G'], 's3://mybucket/1G')
!!!calling func
(D)s4cmd.py:128 >> put_single_file(, , '1G', 's3://mybucket/1G')
!!!calling func
!!!back from func
(D)s4cmd.py:132 << put_single_file(, , '1G', 's3://mybucket/1G'): None
(D)s4cmd.py:128 >> upload(, '1G', 's3://mybucket/1G')
!!!calling func
(D)s4cmd.py:128 >> lookup(, )
!!!calling func
(D)s4cmd.py:404 >> S3APICALL head_object(Bucket='mybucket', Key='1G')
!!!getattr () {'Bucket': 'mybucket', 'Key': '1G'}
!!!back from func
(D)s4cmd.py:132 << lookup(, ): None
(D)s4cmd.py:128 >> read_file_chunk(, '1G', 0, 1073741824)
!!!calling func
!read_file_chunk 1G
!read_file_chunk 1G pos 0
!EXCEPTION: read_file_chunk 1G pos 0:
!!!fail: [Runtime Failure] Unable to read data from source: 1G
(E)s4cmd.py:185 [Runtime Failure] Unable to read data from source: 1G
Exception in thread Thread-1:ad(s)]
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/bin/s4cmd.py", line 530, in run
fail('[Runtime Failure] ', exc_info=e)
File "/usr/bin/s4cmd.py", line 192, in fail
raise RuntimeError(status)
RuntimeError: 1
!!!fail: [Thread Failure] Unable to read data from source: 1G
(E)s4cmd.py:185 [Thread Failure] Unable to read data from source: 1G
!!!fail: [Runtime Exception] 1
(E)s4cmd.py:185 [Runtime Exception] 1
(E)s4cmd.py:187 Traceback (most recent call last):
File "/usr/bin/s4cmd.py", line 1939, in main
CommandHandler(opt).run(args)
File "/usr/bin/s4cmd.py", line 1568, in run
CommandHandler.__dict__[cmd + '_handler'](self, args)
File "/usr/bin/s4cmd.py", line 130, in wrapper
ret = func(*args, **kargs)
File "/usr/bin/s4cmd.py", line 1666, in put_handler
self.s3handler().put_files(source, target)
File "/usr/bin/s4cmd.py", line 130, in wrapper
ret = func(*args, **kargs)
File "/usr/bin/s4cmd.py", line 825, in put_files
pool.join()
File "/usr/bin/s4cmd.py", line 598, in join
self.tasks.join()
File "/usr/bin/s4cmd.py", line 473, in join
fail('[Thread Failure] ', exc_info=self.exc_info)
File "/usr/bin/s4cmd.py", line 192, in fail
raise RuntimeError(status)
RuntimeError: 1
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in s4cmd.py at read_file_chunk(), then follow the thread failure path shown in the traceback through put_files() and ThreadPool.join(). Reproduce the 1GB upload with limited memory and debug output; done means a MemoryError produces a clear, useful failure instead of an inscrutable stack trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100